[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:34:26 dist-test-slave-ppjl krb5kdc[21448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:34:26 dist-test-slave-ppjl krb5kdc[21448](info): set up 2 sockets
Sep 23 17:34:26 dist-test-slave-ppjl krb5kdc[21448](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 17:34:27.033087 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:34:27 dist-test-slave-ppjl krb5kdc[21448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:34:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:34:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:34:27.204036 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41385
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:41385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:27.542728 21461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:27.543047 21461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:27.543205 21461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:27.550740 21461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:34:27.550885 21461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:27.550973 21461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:27.551049 21461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:34:27.551110 21461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:34:27.561520 21461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:41385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:27.562901 21461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:27.564339 21461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:27.576464 21467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:27.576524 21469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:27.576577 21466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:27.577138 21461 server_base.cc:1034] running on GCE node
I20260923 17:34:27.577999 21461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:27.579612 21461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:27.580814 21461 hybrid_clock.cc:648] HybridClock initialized: now 1790184867580756 us; error 52 us; skew 500 ppm
I20260923 17:34:27.581194 21461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:27 dist-test-slave-ppjl krb5kdc[21448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184867, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:27.584292 21461 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:34:27.585740 21461 webserver.cc:460] Webserver started at http://127.20.240.126:46079/ using document root <none> and password file <none>
I20260923 17:34:27.586390 21461 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:27.586483 21461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:27.586743 21461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:27.589695 21461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "fd841e937a654b788380cc25040739ad"
format_stamp: "Formatted at 2026-09-23 17:34:27 on dist-test-slave-ppjl"
server_key: "8f0a50be963c999b965dca084b3b563d"
server_key_iv: "99795baa9db2e2b3babbb79681b636bb"
server_key_version: "encryptionkey@0"
I20260923 17:34:27.590421 21461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "fd841e937a654b788380cc25040739ad"
format_stamp: "Formatted at 2026-09-23 17:34:27 on dist-test-slave-ppjl"
server_key: "8f0a50be963c999b965dca084b3b563d"
server_key_iv: "99795baa9db2e2b3babbb79681b636bb"
server_key_version: "encryptionkey@0"
I20260923 17:34:27.595598 21461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:34:27.598964 21476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:27.600518 21461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:34:27.600692 21461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "fd841e937a654b788380cc25040739ad"
format_stamp: "Formatted at 2026-09-23 17:34:27 on dist-test-slave-ppjl"
server_key: "8f0a50be963c999b965dca084b3b563d"
server_key_iv: "99795baa9db2e2b3babbb79681b636bb"
server_key_version: "encryptionkey@0"
I20260923 17:34:27.600960 21461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:27.634883 21461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:27.639501 21461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:27.639834 21461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:27.657568 21461 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:41385
I20260923 17:34:27.657588 21527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:41385 every 8 connection(s)
I20260923 17:34:27.658999 21461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:34:27.661654 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 21461
I20260923 17:34:27.661994 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:34:27.663780 21528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:27.664561 21441 external_mini_cluster.cc:1428] Setting key a5207a94bc16b3b1bc77e02261117c17
I20260923 17:34:27.675967 21528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad: Bootstrap starting.
I20260923 17:34:27.679273 21528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:27.680696 21528 log.cc:824] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:27.683918 21528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad: No bootstrap required, opened a new log
I20260923 17:34:27.692646 21528 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd841e937a654b788380cc25040739ad" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41385 } }
I20260923 17:34:27.692999 21528 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:27.693099 21528 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd841e937a654b788380cc25040739ad, State: Initialized, Role: FOLLOWER
I20260923 17:34:27.693696 21528 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd841e937a654b788380cc25040739ad" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41385 } }
I20260923 17:34:27.693917 21528 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:34:27.694051 21528 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:34:27.694224 21528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [term 0 FOLLOWER]: Advancing to term 1
Sep 23 17:34:27 dist-test-slave-ppjl krb5kdc[21448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:27.696916 21528 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd841e937a654b788380cc25040739ad" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41385 } }
I20260923 17:34:27.697569 21528 leader_election.cc:302] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd841e937a654b788380cc25040739ad; no voters:
I20260923 17:34:27.698300 21528 leader_election.cc:288] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:34:27.698400 21533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:27.700861 21533 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [term 1 LEADER]: Becoming Leader. State: Replica: fd841e937a654b788380cc25040739ad, State: Running, Role: LEADER
I20260923 17:34:27.701082 21528 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:34:27.701781 21533 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd841e937a654b788380cc25040739ad" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41385 } }
I20260923 17:34:27.706607 21534 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd841e937a654b788380cc25040739ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd841e937a654b788380cc25040739ad" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41385 } } }
I20260923 17:34:27.707319 21534 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:27.709353 21541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:34:27.711143 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:27.669253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.20.240.126:41385)
0923 17:34:27.670163 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:27.670180 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:27.671541 (+ 1361us) server_negotiation.cc:407] Connection header received
0923 17:34:27.680924 (+ 9383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:27.680954 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:27.681579 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:27.681952 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:27.684912 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:27.687002 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:27.688744 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:27.689948 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:27.690141 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:27.696072 (+ 5931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:27.696083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:27.696102 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:27.696156 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:27.700252 (+ 4096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:27.701449 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:27.701458 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:27.701467 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:27.701801 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:27.702439 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:27.702444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:27.702447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:27.703400 (+ 953us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:27.703853 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:27.707551 (+ 3698us) server_negotiation.cc:299] Negotiation successful
0923 17:34:27.708097 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":380,"threads_started":1}
I20260923 17:34:27.713426 21535 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd841e937a654b788380cc25040739ad. Latest consensus state: current_term: 1 leader_uuid: "fd841e937a654b788380cc25040739ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd841e937a654b788380cc25040739ad" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41385 } } }
I20260923 17:34:27.714110 21535 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:27.723898 21541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:34:27.734900 21541 catalog_manager.cc:1347] Generated new cluster ID: 27676784cad04d4aa01c474e34390e0f
I20260923 17:34:27.735050 21541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:34:27.750049 21541 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:34:27.752033 21541 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:34:27.772535 21541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd841e937a654b788380cc25040739ad: Generated new TSK 0
I20260923 17:34:27.773712 21541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:34:27.831873 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41385
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:28.185844 21554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:28.186128 21554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:28.186223 21554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:28.193493 21554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:28.193620 21554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:28.193743 21554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:34:28.203806 21554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41385
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:28.205170 21554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:28.206615 21554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:28.220180 21560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:28.220180 21562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:28.220155 21559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:28.221426 21554 server_base.cc:1034] running on GCE node
I20260923 17:34:28.222054 21554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:28.223006 21554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:28.224218 21554 hybrid_clock.cc:648] HybridClock initialized: now 1790184868224156 us; error 52 us; skew 500 ppm
I20260923 17:34:28.224607 21554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:28 dist-test-slave-ppjl krb5kdc[21448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184868, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:28.227782 21554 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:34:28.229307 21554 webserver.cc:460] Webserver started at http://127.20.240.65:35565/ using document root <none> and password file <none>
I20260923 17:34:28.230238 21554 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:28.230357 21554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:28.230643 21554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:28.233633 21554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "24bfa2da19e74f7d8e74268f536a9e7d"
format_stamp: "Formatted at 2026-09-23 17:34:28 on dist-test-slave-ppjl"
server_key: "e40c789cdf3609629b376f7141060ae3"
server_key_iv: "fa4f22ae9bd1c00b0471c6c24c242ccd"
server_key_version: "encryptionkey@0"
I20260923 17:34:28.234364 21554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "24bfa2da19e74f7d8e74268f536a9e7d"
format_stamp: "Formatted at 2026-09-23 17:34:28 on dist-test-slave-ppjl"
server_key: "e40c789cdf3609629b376f7141060ae3"
server_key_iv: "fa4f22ae9bd1c00b0471c6c24c242ccd"
server_key_version: "encryptionkey@0"
I20260923 17:34:28.239534 21554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260923 17:34:28.243526 21569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:28.245018 21554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:34:28.245175 21554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "24bfa2da19e74f7d8e74268f536a9e7d"
format_stamp: "Formatted at 2026-09-23 17:34:28 on dist-test-slave-ppjl"
server_key: "e40c789cdf3609629b376f7141060ae3"
server_key_iv: "fa4f22ae9bd1c00b0471c6c24c242ccd"
server_key_version: "encryptionkey@0"
I20260923 17:34:28.245368 21554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:28.257752 21554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:28.262230 21554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:28.262492 21554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:28.263486 21554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:34:28.264890 21554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:28.264966 21554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:28.265059 21554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:28.265103 21554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:28.296679 21554 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:41983
I20260923 17:34:28.296717 21681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:41983 every 8 connection(s)
I20260923 17:34:28.297937 21554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:34:28.302207 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 21554
I20260923 17:34:28.302455 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:34:28.303251 21441 external_mini_cluster.cc:1428] Setting key ce2652b6f51c2348b11d455b6b2c20c9
Sep 23 17:34:28 dist-test-slave-ppjl krb5kdc[21448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184868, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
I20260923 17:34:28.320731 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:28.300660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:47739 (local address 127.20.240.126:41385)
0923 17:34:28.301142 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:28.301146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:28.302465 (+ 1319us) server_negotiation.cc:407] Connection header received
0923 17:34:28.304916 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:28.304921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:28.305011 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:28.305148 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:28.306778 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:28.307816 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:28.309519 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:28.309919 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:28.310024 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:28.314087 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:28.314092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:28.314095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:28.314130 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:28.317327 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:28.318194 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:28.318201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:28.318205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:28.318358 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:28.318849 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:28.318854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:28.318858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:28.319252 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:28.319406 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:28.320228 (+ 822us) server_negotiation.cc:299] Negotiation successful
0923 17:34:28.320442 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":239,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
I20260923 17:34:28.321784 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:28.301252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41385 (local address 127.20.240.65:47739)
0923 17:34:28.302180 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:28.302235 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:28.304633 (+ 2398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:28.305443 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:28.305452 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:28.305994 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:28.306593 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:28.306611 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:28.307969 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:28.307974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:28.309368 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:28.309391 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:28.310151 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:28.310159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:28.310455 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:28.311180 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:28.311209 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:28.313847 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:28.317511 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:28.317520 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:28.317535 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:28.318059 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:28.318500 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:28.318504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:28.318507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:28.318724 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:28.319382 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:28.319389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:28.319801 (+ 412us) client_negotiation.cc:769] Sending connection context
0923 17:34:28.320078 (+ 277us) client_negotiation.cc:240] Negotiation successful
0923 17:34:28.320563 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":371,"threads_started":1}
I20260923 17:34:28.323379 21682 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41385
I20260923 17:34:28.323678 21682 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:28.324373 21682 heartbeater.cc:511] Master 127.20.240.126:41385 requested a full tablet report, sending...
I20260923 17:34:28.326409 21493 ts_manager.cc:194] Registered new tserver with Master: 24bfa2da19e74f7d8e74268f536a9e7d (127.20.240.65:41983)
I20260923 17:34:28.327916 21493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:47739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:34:28.343007 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41385
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:28.689853 21688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:28.690155 21688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:28.690284 21688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:28.697889 21688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:28.698020 21688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:28.698187 21688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:34:28.708590 21688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41385
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:28.709959 21688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:28.711398 21688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:28.725261 21693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:28.725181 21696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:28.725203 21694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:28.733064 21688 server_base.cc:1034] running on GCE node
I20260923 17:34:28.733793 21688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:28.734835 21688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:28.736047 21688 hybrid_clock.cc:648] HybridClock initialized: now 1790184868735996 us; error 48 us; skew 500 ppm
I20260923 17:34:28.736375 21688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:28 dist-test-slave-ppjl krb5kdc[21448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184868, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:28.739768 21688 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:34:28.741451 21688 webserver.cc:460] Webserver started at http://127.20.240.66:42153/ using document root <none> and password file <none>
I20260923 17:34:28.742209 21688 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:28.742327 21688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:28.742723 21688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:28.745887 21688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "70d3186a91714202a46de1d4bb5f2c08"
format_stamp: "Formatted at 2026-09-23 17:34:28 on dist-test-slave-ppjl"
server_key: "8a10ad188364274158bbfe641dcb46e0"
server_key_iv: "c53f2c9b2d866888b6ea9dbc129bf8b5"
server_key_version: "encryptionkey@0"
I20260923 17:34:28.746682 21688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "70d3186a91714202a46de1d4bb5f2c08"
format_stamp: "Formatted at 2026-09-23 17:34:28 on dist-test-slave-ppjl"
server_key: "8a10ad188364274158bbfe641dcb46e0"
server_key_iv: "c53f2c9b2d866888b6ea9dbc129bf8b5"
server_key_version: "encryptionkey@0"
I20260923 17:34:28.752437 21688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:34:28.756297 21703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:28.757972 21688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:34:28.758154 21688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "70d3186a91714202a46de1d4bb5f2c08"
format_stamp: "Formatted at 2026-09-23 17:34:28 on dist-test-slave-ppjl"
server_key: "8a10ad188364274158bbfe641dcb46e0"
server_key_iv: "c53f2c9b2d866888b6ea9dbc129bf8b5"
server_key_version: "encryptionkey@0"
I20260923 17:34:28.758361 21688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:28.773234 21688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:28.777791 21688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:28.778059 21688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:28.779119 21688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:34:28.780606 21688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:28.780683 21688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:28.780771 21688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:28.780845 21688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:28.813067 21688 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:36225
I20260923 17:34:28.813122 21815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:36225 every 8 connection(s)
I20260923 17:34:28.814384 21688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:34:28.820406 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 21688
I20260923 17:34:28.820637 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:34:28.821182 21441 external_mini_cluster.cc:1428] Setting key a03a8732a94e0d6b7291d44e37e16cca
Sep 23 17:34:28 dist-test-slave-ppjl krb5kdc[21448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184868, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
I20260923 17:34:28.835477 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:28.817139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:53573 (local address 127.20.240.126:41385)
0923 17:34:28.817337 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:28.817343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:28.818583 (+ 1240us) server_negotiation.cc:407] Connection header received
0923 17:34:28.820747 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:28.820754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:28.820846 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:28.820950 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:28.822851 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:28.823643 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:28.825271 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:28.825692 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:28.825805 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:28.829629 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:28.829635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:28.829638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:28.829676 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:28.832183 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:28.833001 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:28.833005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:28.833007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:28.833140 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:28.833593 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:28.833597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:28.833599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:28.833889 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:28.834026 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:28.834842 (+ 816us) server_negotiation.cc:299] Negotiation successful
0923 17:34:28.835099 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:34:28.836546 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:28.817676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41385 (local address 127.20.240.66:53573)
0923 17:34:28.818395 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:28.818457 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:28.820423 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:28.821241 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:28.821253 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:28.821955 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:28.822678 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:28.822699 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:28.823798 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:28.823802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:28.825126 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:28.825139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:28.825912 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:28.825920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:28.826448 (+ 528us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:28.827111 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:28.827136 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:28.829359 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:28.832367 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:28.832376 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:28.832392 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:28.832871 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:28.833252 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:28.833257 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:28.833261 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:28.833481 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:28.834018 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:28.834026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:28.834419 (+ 393us) client_negotiation.cc:769] Sending connection context
0923 17:34:28.834712 (+ 293us) client_negotiation.cc:240] Negotiation successful
0923 17:34:28.835358 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":274,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
I20260923 17:34:28.838129 21816 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41385
I20260923 17:34:28.838483 21816 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:28.839298 21816 heartbeater.cc:511] Master 127.20.240.126:41385 requested a full tablet report, sending...
I20260923 17:34:28.840894 21493 ts_manager.cc:194] Registered new tserver with Master: 70d3186a91714202a46de1d4bb5f2c08 (127.20.240.66:36225)
I20260923 17:34:28.841758 21493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:53573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:34:28.858712 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41385
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:44757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:29.207918 21821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:29.208211 21821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:29.208312 21821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:29.215610 21821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:29.215739 21821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:29.215878 21821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:34:29.226128 21821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41385
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:29.227591 21821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:29.228919 21821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:29.242033 21827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:29.242079 21829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:29.242115 21826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:29.242674 21821 server_base.cc:1034] running on GCE node
I20260923 17:34:29.243443 21821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:29.244531 21821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:29.245748 21821 hybrid_clock.cc:648] HybridClock initialized: now 1790184869245696 us; error 41 us; skew 500 ppm
I20260923 17:34:29.246096 21821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:29 dist-test-slave-ppjl krb5kdc[21448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184869, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:29.249509 21821 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:34:29.251106 21821 webserver.cc:460] Webserver started at http://127.20.240.67:43785/ using document root <none> and password file <none>
I20260923 17:34:29.251807 21821 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:29.251937 21821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:29.252235 21821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:29.255235 21821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "25a6dcff20f74a9bb236c47217eadf2e"
format_stamp: "Formatted at 2026-09-23 17:34:29 on dist-test-slave-ppjl"
server_key: "b41732232eb5f14e9457be13a7d0cbbc"
server_key_iv: "440f90abca6a13fb5867d5c2981cb492"
server_key_version: "encryptionkey@0"
I20260923 17:34:29.256035 21821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "25a6dcff20f74a9bb236c47217eadf2e"
format_stamp: "Formatted at 2026-09-23 17:34:29 on dist-test-slave-ppjl"
server_key: "b41732232eb5f14e9457be13a7d0cbbc"
server_key_iv: "440f90abca6a13fb5867d5c2981cb492"
server_key_version: "encryptionkey@0"
I20260923 17:34:29.261255 21821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:34:29.264930 21836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:29.266335 21821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:34:29.266502 21821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "25a6dcff20f74a9bb236c47217eadf2e"
format_stamp: "Formatted at 2026-09-23 17:34:29 on dist-test-slave-ppjl"
server_key: "b41732232eb5f14e9457be13a7d0cbbc"
server_key_iv: "440f90abca6a13fb5867d5c2981cb492"
server_key_version: "encryptionkey@0"
I20260923 17:34:29.266693 21821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:29.288244 21821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:29.292609 21821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:29.292922 21821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:29.293879 21821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:34:29.295328 21821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:29.295409 21821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:29.295535 21821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:29.295588 21821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:29.328720 21821 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:34599
I20260923 17:34:29.328764 21948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:34599 every 8 connection(s)
I20260923 17:34:29.330046 21821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:34:29.330878 21682 heartbeater.cc:503] Master 127.20.240.126:41385 was elected leader, sending a full tablet report...
I20260923 17:34:29.336360 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 21821
I20260923 17:34:29.336580 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:34:29.337100 21441 external_mini_cluster.cc:1428] Setting key 9e3d1809049fdb64be7d94398dfae196
Sep 23 17:34:29 dist-test-slave-ppjl krb5kdc[21448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184869, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:29.351737 21684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:29.333167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:54025 (local address 127.20.240.126:41385)
0923 17:34:29.333378 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:29.333384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:29.334658 (+ 1274us) server_negotiation.cc:407] Connection header received
0923 17:34:29.336826 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:29.336830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:29.336919 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:29.337041 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:29.338930 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:29.339951 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:29.341803 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:29.342245 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:29.342337 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:29.345810 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:29.345816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:29.345819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:29.345855 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:29.348541 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:29.349288 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:29.349293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:29.349365 (+ 72us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:29.349508 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:29.350010 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:29.350014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:29.350016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:29.350364 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:29.350542 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:29.351252 (+ 710us) server_negotiation.cc:299] Negotiation successful
0923 17:34:29.351429 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 17:34:29.352936 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:29.333743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41385 (local address 127.20.240.67:54025)
0923 17:34:29.334490 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:29.334530 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:29.336504 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:29.337333 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:29.337345 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:29.338143 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:29.338759 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:29.338778 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:29.340107 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:29.340112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:29.341664 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:29.341678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:29.342436 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:29.342441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:29.342668 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:29.343316 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:29.343343 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:29.345572 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:29.348690 (+ 3118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:29.348697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:29.348713 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:29.349173 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:29.349649 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:29.349653 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:29.349656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:29.349868 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:29.350490 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:29.350499 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:29.350860 (+ 361us) client_negotiation.cc:769] Sending connection context
0923 17:34:29.351116 (+ 256us) client_negotiation.cc:240] Negotiation successful
0923 17:34:29.351655 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":293,"threads_started":1}
I20260923 17:34:29.354406 21949 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41385
I20260923 17:34:29.354894 21949 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:29.355710 21949 heartbeater.cc:511] Master 127.20.240.126:41385 requested a full tablet report, sending...
I20260923 17:34:29.357384 21493 ts_manager.cc:194] Registered new tserver with Master: 25a6dcff20f74a9bb236c47217eadf2e (127.20.240.67:34599)
I20260923 17:34:29.358247 21493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:54025
I20260923 17:34:29.366323 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:34:29 dist-test-slave-ppjl krb5kdc[21448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184869, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:34:29 dist-test-slave-ppjl krb5kdc[21448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184869, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:34:29.415234 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:29.397112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.20.240.65:41983)
0923 17:34:29.398646 (+ 1534us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:29.398653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:29.398681 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:34:29.398747 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:29.398754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:29.398973 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:29.399112 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:29.400830 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:29.402074 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:29.403676 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:29.404296 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:29.404418 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:29.408748 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:29.408763 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:29.408780 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:29.408830 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:29.412071 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:29.412921 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:29.412929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:29.412931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:29.413097 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:29.413550 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:29.413554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:29.413557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:29.413940 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:29.414237 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:29.414582 (+ 345us) server_negotiation.cc:299] Negotiation successful
0923 17:34:29.414849 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1366,"thread_start_us":1196,"threads_started":1}
W20260923 17:34:29.416616 21597 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53714
Sep 23 17:34:29 dist-test-slave-ppjl krb5kdc[21448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:34:29 dist-test-slave-ppjl krb5kdc[21448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:34:29.460129 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:29.445484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.20.240.65:41983)
0923 17:34:29.445759 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:29.445772 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:29.445944 (+ 172us) server_negotiation.cc:407] Connection header received
0923 17:34:29.446081 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:29.446086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:29.446153 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:29.446240 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:29.447623 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:29.448304 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:29.449611 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:29.450112 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:29.450210 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:29.454296 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:29.454301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:29.454305 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:29.454352 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:29.457255 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:29.458091 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:29.458096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:29.458098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:29.458254 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:29.458683 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:29.458686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:29.458689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:29.459005 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:29.459288 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:29.459640 (+ 352us) server_negotiation.cc:299] Negotiation successful
0923 17:34:29.459829 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260923 17:34:29.461867 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 21554
I20260923 17:34:29.475701 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:29.463021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53740 (local address 127.20.240.65:41983)
0923 17:34:29.463248 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:29.463254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:29.463468 (+ 214us) server_negotiation.cc:407] Connection header received
0923 17:34:29.463627 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:29.463631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:29.463687 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:29.463764 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:29.465006 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:29.465701 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:29.466965 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:29.467485 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:29.467596 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:29.469864 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:29.469869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:29.469872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:29.469908 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:29.472952 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:29.473714 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:29.473718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:29.473721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:29.473863 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:29.474255 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:29.474259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:29.474261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:29.474540 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:29.474759 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:29.475100 (+ 341us) server_negotiation.cc:299] Negotiation successful
0923 17:34:29.475345 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:34:29.476418 21677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:34:29.559478 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 21554
I20260923 17:34:29.573180 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 21688
Sep 23 17:34:29 dist-test-slave-ppjl krb5kdc[21448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:34:29.590298 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:29.574123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45420 (local address 127.20.240.66:36225)
0923 17:34:29.574649 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:29.574656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:29.574683 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:34:29.574745 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:29.574752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:29.574964 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:29.575147 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:29.576480 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:29.577521 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:29.578836 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:29.579507 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:29.579639 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:29.584259 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:29.584276 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:29.584295 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:29.584340 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:29.587220 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:29.587998 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:29.588007 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:29.588009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:29.588147 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:29.588606 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:29.588610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:29.588612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:29.588989 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:29.589311 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:29.589665 (+ 354us) server_negotiation.cc:299] Negotiation successful
0923 17:34:29.589963 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":271,"threads_started":1}
I20260923 17:34:29.591609 21811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:34:29.678992 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 21688
I20260923 17:34:29.697620 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 21821
Sep 23 17:34:29 dist-test-slave-ppjl krb5kdc[21448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:34:29.715953 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:29.698727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.20.240.67:34599)
0923 17:34:29.699544 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:29.699552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:29.699581 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:34:29.699646 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:29.699653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:29.699872 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:29.700018 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:29.701398 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:29.702610 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:29.704129 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:29.704818 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:29.704921 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:29.709636 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:29.709654 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:29.709672 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:29.709720 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:29.712724 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:29.713662 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:29.713669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:29.713672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:29.713809 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:29.714243 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:29.714246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:29.714249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:29.714636 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:29.714875 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:29.715294 (+ 419us) server_negotiation.cc:299] Negotiation successful
0923 17:34:29.715573 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":444,"threads_started":1}
I20260923 17:34:29.717319 21944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:34:29.800312 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 21821
I20260923 17:34:29.813630 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 21461
I20260923 17:34:29.814815 21523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:34:29.918520 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 21461
2026-09-23T17:34:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3902 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:34:30 dist-test-slave-ppjl krb5kdc[21970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:34:30 dist-test-slave-ppjl krb5kdc[21970](info): set up 2 sockets
Sep 23 17:34:30 dist-test-slave-ppjl krb5kdc[21970](info): commencing operation
krb5kdc: starting...
W20260923 17:34:30.927810 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:34:31 dist-test-slave-ppjl krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:34:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:34:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:34:31.107450 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:35799
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:35799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:31.428138 21983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:31.428488 21983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:31.428614 21983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:31.436210 21983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:34:31.436350 21983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:31.436427 21983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:31.436501 21983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:34:31.436563 21983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:34:31.446738 21983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:35799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:35799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:31.448122 21983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:31.449568 21983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:31.461493 21988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:31.461432 21991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:31.461515 21989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:31.462092 21983 server_base.cc:1034] running on GCE node
I20260923 17:34:31.462884 21983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:31.464465 21983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:31.465674 21983 hybrid_clock.cc:648] HybridClock initialized: now 1790184871465630 us; error 34 us; skew 500 ppm
I20260923 17:34:31.466032 21983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:31 dist-test-slave-ppjl krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184871, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:31.473014 21983 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:34:31.474745 21983 webserver.cc:460] Webserver started at http://127.20.240.126:39647/ using document root <none> and password file <none>
I20260923 17:34:31.475550 21983 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:31.475668 21983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:31.475996 21983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:31.479120 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "76cdcaf4bd2c4c1f91a1d638583df520"
format_stamp: "Formatted at 2026-09-23 17:34:31 on dist-test-slave-ppjl"
server_key: "9bccddaec345d532c72fb3dc1400ce59"
server_key_iv: "5399a960fc2fe88b6d8f1c61d66e841e"
server_key_version: "encryptionkey@0"
I20260923 17:34:31.480068 21983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "76cdcaf4bd2c4c1f91a1d638583df520"
format_stamp: "Formatted at 2026-09-23 17:34:31 on dist-test-slave-ppjl"
server_key: "9bccddaec345d532c72fb3dc1400ce59"
server_key_iv: "5399a960fc2fe88b6d8f1c61d66e841e"
server_key_version: "encryptionkey@0"
I20260923 17:34:31.486100 21983 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 17:34:31.489955 21998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:31.491623 21983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:34:31.491808 21983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "76cdcaf4bd2c4c1f91a1d638583df520"
format_stamp: "Formatted at 2026-09-23 17:34:31 on dist-test-slave-ppjl"
server_key: "9bccddaec345d532c72fb3dc1400ce59"
server_key_iv: "5399a960fc2fe88b6d8f1c61d66e841e"
server_key_version: "encryptionkey@0"
I20260923 17:34:31.492014 21983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:31.514562 21983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:31.519309 21983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:31.519613 21983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:31.538383 21983 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:35799
I20260923 17:34:31.538419 22049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:35799 every 8 connection(s)
I20260923 17:34:31.539852 21983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:34:31.543712 22050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:31.545253 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 21983
I20260923 17:34:31.545498 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:34:31.546294 21441 external_mini_cluster.cc:1428] Setting key b1e6f784e96fff18ed0599f63e2ae473
I20260923 17:34:31.558276 22050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520: Bootstrap starting.
Sep 23 17:34:31 dist-test-slave-ppjl krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:31.562194 22050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:31.563612 22050 log.cc:824] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:31.567400 22050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520: No bootstrap required, opened a new log
I20260923 17:34:31.571978 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:31.548417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.20.240.126:35799)
0923 17:34:31.549205 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:31.549217 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:31.549259 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:34:31.550903 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:31.550932 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:31.551406 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:31.551744 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:31.553594 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:31.554954 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:31.556780 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:31.557517 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:31.557660 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:31.562478 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:31.562490 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:31.562512 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:31.562583 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:31.566513 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:31.567404 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:31.567414 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:31.567423 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:31.567642 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:31.568141 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:31.568146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:31.568150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:31.568912 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:31.569220 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:31.569860 (+ 640us) server_negotiation.cc:299] Negotiation successful
0923 17:34:31.570389 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":352,"threads_started":1}
I20260923 17:34:31.576440 22050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdcaf4bd2c4c1f91a1d638583df520" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35799 } }
I20260923 17:34:31.576786 22050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:31.576857 22050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76cdcaf4bd2c4c1f91a1d638583df520, State: Initialized, Role: FOLLOWER
I20260923 17:34:31.577374 22050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdcaf4bd2c4c1f91a1d638583df520" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35799 } }
I20260923 17:34:31.577592 22050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:34:31.577687 22050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:34:31.577809 22050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:31.579953 22050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdcaf4bd2c4c1f91a1d638583df520" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35799 } }
I20260923 17:34:31.580407 22050 leader_election.cc:302] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76cdcaf4bd2c4c1f91a1d638583df520; no voters:
I20260923 17:34:31.581043 22050 leader_election.cc:288] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:34:31.581164 22055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:31.582201 22055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [term 1 LEADER]: Becoming Leader. State: Replica: 76cdcaf4bd2c4c1f91a1d638583df520, State: Running, Role: LEADER
I20260923 17:34:31.582779 22055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdcaf4bd2c4c1f91a1d638583df520" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35799 } }
I20260923 17:34:31.583405 22050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:34:31.585719 22057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76cdcaf4bd2c4c1f91a1d638583df520. Latest consensus state: current_term: 1 leader_uuid: "76cdcaf4bd2c4c1f91a1d638583df520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdcaf4bd2c4c1f91a1d638583df520" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35799 } } }
I20260923 17:34:31.586171 22057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:31.586404 22056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76cdcaf4bd2c4c1f91a1d638583df520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76cdcaf4bd2c4c1f91a1d638583df520" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35799 } } }
I20260923 17:34:31.587253 22056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520 [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:31.589570 22062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:34:31.594239 22062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:34:31.603799 22062 catalog_manager.cc:1347] Generated new cluster ID: c9fdf5f95b814719bcfc1ba4145c9df0
I20260923 17:34:31.603935 22062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:34:31.628386 22062 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:34:31.629921 22062 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:34:31.647248 22062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76cdcaf4bd2c4c1f91a1d638583df520: Generated new TSK 0
I20260923 17:34:31.648231 22062 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:34:31.694860 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:35799
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:32.041958 22076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:32.042271 22076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:32.042423 22076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:32.049950 22076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:32.050081 22076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:32.050237 22076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:34:32.061105 22076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:35799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:32.062492 22076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:32.063951 22076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:32.077430 22084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:32.077453 22082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:32.077559 22081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:32.078274 22076 server_base.cc:1034] running on GCE node
I20260923 17:34:32.078934 22076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:32.079989 22076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:32.081207 22076 hybrid_clock.cc:648] HybridClock initialized: now 1790184872081153 us; error 48 us; skew 500 ppm
I20260923 17:34:32.081636 22076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:32 dist-test-slave-ppjl krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184872, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:32.085069 22076 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:34:32.086796 22076 webserver.cc:460] Webserver started at http://127.20.240.65:41385/ using document root <none> and password file <none>
I20260923 17:34:32.087563 22076 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:32.087672 22076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:32.087957 22076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:32.091243 22076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "be36632deef7488e9d95aed20a78a3d7"
format_stamp: "Formatted at 2026-09-23 17:34:32 on dist-test-slave-ppjl"
server_key: "26e96572cbfe8adb4bb76315f903e4e6"
server_key_iv: "214a70e195598c7d41b092173649d04e"
server_key_version: "encryptionkey@0"
I20260923 17:34:32.092061 22076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "be36632deef7488e9d95aed20a78a3d7"
format_stamp: "Formatted at 2026-09-23 17:34:32 on dist-test-slave-ppjl"
server_key: "26e96572cbfe8adb4bb76315f903e4e6"
server_key_iv: "214a70e195598c7d41b092173649d04e"
server_key_version: "encryptionkey@0"
I20260923 17:34:32.097745 22076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260923 17:34:32.101593 22091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:32.103060 22076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 17:34:32.103267 22076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "be36632deef7488e9d95aed20a78a3d7"
format_stamp: "Formatted at 2026-09-23 17:34:32 on dist-test-slave-ppjl"
server_key: "26e96572cbfe8adb4bb76315f903e4e6"
server_key_iv: "214a70e195598c7d41b092173649d04e"
server_key_version: "encryptionkey@0"
I20260923 17:34:32.103521 22076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:32.129616 22076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:32.134286 22076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:32.134557 22076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:32.135528 22076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:34:32.137162 22076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:32.137253 22076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:32.137380 22076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:32.137451 22076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:32.169132 22076 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:37997
I20260923 17:34:32.169176 22203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:37997 every 8 connection(s)
I20260923 17:34:32.170396 22076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:34:32.174782 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 22076
I20260923 17:34:32.174990 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:34:32.175616 21441 external_mini_cluster.cc:1428] Setting key 0cc34f58e1d4a0f1619d493fd329cecc
Sep 23 17:34:32 dist-test-slave-ppjl krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184872, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
I20260923 17:34:32.191212 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:32.172996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:43285 (local address 127.20.240.126:35799)
0923 17:34:32.173565 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:32.173570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:32.174447 (+ 877us) server_negotiation.cc:407] Connection header received
0923 17:34:32.176688 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:32.176693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:32.176778 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:32.176870 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:32.178577 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:32.179394 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:32.180979 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:32.181408 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:32.181524 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:32.185312 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:32.185317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:32.185321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:32.185362 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:32.187973 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:32.188719 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:32.188724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:32.188726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:32.188855 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:32.189300 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:32.189304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:32.189307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:32.189592 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:32.189730 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:32.190569 (+ 839us) server_negotiation.cc:299] Negotiation successful
0923 17:34:32.190809 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":265,"threads_started":1}
I20260923 17:34:32.192230 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:32.173434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35799 (local address 127.20.240.65:43285)
0923 17:34:32.174244 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:32.174292 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:32.176408 (+ 2116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:32.177145 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:32.177155 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:32.177805 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:32.178406 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:32.178424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:32.179540 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:32.179544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:32.180815 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:32.180833 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:32.181647 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:32.181653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:32.181889 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:32.182572 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:32.182597 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:32.185082 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:32.188127 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:32.188137 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:32.188157 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:32.188590 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:32.188972 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:32.188978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:32.188982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:32.189178 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:32.189730 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:32.189739 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:32.190154 (+ 415us) client_negotiation.cc:769] Sending connection context
0923 17:34:32.190412 (+ 258us) client_negotiation.cc:240] Negotiation successful
0923 17:34:32.190993 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":358,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
I20260923 17:34:32.193894 22204 heartbeater.cc:347] Connected to a master server at 127.20.240.126:35799
I20260923 17:34:32.194227 22204 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:32.194955 22204 heartbeater.cc:511] Master 127.20.240.126:35799 requested a full tablet report, sending...
I20260923 17:34:32.197408 22015 ts_manager.cc:194] Registered new tserver with Master: be36632deef7488e9d95aed20a78a3d7 (127.20.240.65:37997)
I20260923 17:34:32.199472 22015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:43285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:34:32.215416 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:35799
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:32.530804 22210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:32.531116 22210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:32.531241 22210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:32.539636 22210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:32.539786 22210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:32.539974 22210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:34:32.555656 22210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:35799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:32.557402 22210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:32.559130 22210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:32.572830 22215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:32.572892 22216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:32.572855 22218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:32.577008 22210 server_base.cc:1034] running on GCE node
I20260923 17:34:32.577711 22210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:32.578698 22210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:32.579900 22210 hybrid_clock.cc:648] HybridClock initialized: now 1790184872579843 us; error 53 us; skew 500 ppm
I20260923 17:34:32.580224 22210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:32 dist-test-slave-ppjl krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184872, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:32.583328 22210 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:34:32.584847 22210 webserver.cc:460] Webserver started at http://127.20.240.66:39243/ using document root <none> and password file <none>
I20260923 17:34:32.585523 22210 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:32.585623 22210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:32.585876 22210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:32.588910 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "08969e2e21644610bfd47eb5acb19481"
format_stamp: "Formatted at 2026-09-23 17:34:32 on dist-test-slave-ppjl"
server_key: "edd44d0dc4e12ff93baef8a65f95812d"
server_key_iv: "1d8dcb5c71314352fb031eafbd9288ed"
server_key_version: "encryptionkey@0"
I20260923 17:34:32.589746 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "08969e2e21644610bfd47eb5acb19481"
format_stamp: "Formatted at 2026-09-23 17:34:32 on dist-test-slave-ppjl"
server_key: "edd44d0dc4e12ff93baef8a65f95812d"
server_key_iv: "1d8dcb5c71314352fb031eafbd9288ed"
server_key_version: "encryptionkey@0"
I20260923 17:34:32.595124 22210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 17:34:32.598922 22225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:32.600478 22210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:34:32.600634 22210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "08969e2e21644610bfd47eb5acb19481"
format_stamp: "Formatted at 2026-09-23 17:34:32 on dist-test-slave-ppjl"
server_key: "edd44d0dc4e12ff93baef8a65f95812d"
server_key_iv: "1d8dcb5c71314352fb031eafbd9288ed"
server_key_version: "encryptionkey@0"
I20260923 17:34:32.600836 22210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:32.615612 22210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:32.619764 22210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:32.620059 22210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:32.621147 22210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:34:32.622545 22210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:32.622640 22210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:32.622766 22210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:32.622823 22210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:32.652918 22210 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:34083
I20260923 17:34:32.652951 22337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:34083 every 8 connection(s)
I20260923 17:34:32.654155 22210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:34:32.661947 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 22210
I20260923 17:34:32.662107 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:34:32.662573 21441 external_mini_cluster.cc:1428] Setting key c7fe6727eecb05d31184d28c75bfab07
Sep 23 17:34:32 dist-test-slave-ppjl krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184872, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:32.674599 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:32.656745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:33323 (local address 127.20.240.126:35799)
0923 17:34:32.656929 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:32.656933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:32.658052 (+ 1119us) server_negotiation.cc:407] Connection header received
0923 17:34:32.659918 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:32.659924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:32.660008 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:32.660092 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:32.661624 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:32.662394 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:32.664404 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:32.664985 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:32.665112 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:32.668705 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:32.668710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:32.668714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:32.668747 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:32.671277 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:32.672097 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:32.672103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:32.672107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:32.672265 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:32.672789 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:32.672793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:32.672796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:32.673137 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:32.673278 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:32.673978 (+ 700us) server_negotiation.cc:299] Negotiation successful
0923 17:34:32.674228 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:34:32.675549 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:32.657170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35799 (local address 127.20.240.66:33323)
0923 17:34:32.657844 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:32.657916 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:32.659646 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:32.660332 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:32.660341 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:32.660846 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:32.661451 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:32.661468 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:32.662546 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:32.662550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:32.664234 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:32.664247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:32.665206 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:32.665211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:32.665425 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:32.666121 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:32.666144 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:32.668446 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:32.671436 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:32.671447 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:32.671467 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:32.671909 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:32.672398 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:32.672403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:32.672407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:32.672668 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:32.673250 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:32.673256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:32.673603 (+ 347us) client_negotiation.cc:769] Sending connection context
0923 17:34:32.673826 (+ 223us) client_negotiation.cc:240] Negotiation successful
0923 17:34:32.674245 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":258,"threads_started":1}
I20260923 17:34:32.677093 22338 heartbeater.cc:347] Connected to a master server at 127.20.240.126:35799
I20260923 17:34:32.677443 22338 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
I20260923 17:34:32.678246 22338 heartbeater.cc:511] Master 127.20.240.126:35799 requested a full tablet report, sending...
I20260923 17:34:32.679803 22015 ts_manager.cc:194] Registered new tserver with Master: 08969e2e21644610bfd47eb5acb19481 (127.20.240.66:34083)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
I20260923 17:34:32.680698 22015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:33323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:34:32.704205 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:35799
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:33.033329 22343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:33.033645 22343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:33.033769 22343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:33.041658 22343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:33.041795 22343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:33.041946 22343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:34:33.054136 22343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:35799
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:33.055941 22343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:33.057626 22343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:33.072146 22351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:33.072252 22349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:33.072153 22348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:33.072997 22343 server_base.cc:1034] running on GCE node
I20260923 17:34:33.073642 22343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:33.074677 22343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:33.075917 22343 hybrid_clock.cc:648] HybridClock initialized: now 1790184873075842 us; error 61 us; skew 500 ppm
I20260923 17:34:33.076253 22343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:33 dist-test-slave-ppjl krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184873, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:33.079556 22343 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:34:33.081120 22343 webserver.cc:460] Webserver started at http://127.20.240.67:44665/ using document root <none> and password file <none>
I20260923 17:34:33.081782 22343 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:33.081879 22343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:33.082140 22343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:33.085168 22343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "c611c3b9b32b47e891747d60488e7320"
format_stamp: "Formatted at 2026-09-23 17:34:33 on dist-test-slave-ppjl"
server_key: "f420d5b16ecf3e8ef3c096c7e81a2caf"
server_key_iv: "2a0fb2b976477e43c3534284916805ef"
server_key_version: "encryptionkey@0"
I20260923 17:34:33.085928 22343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "c611c3b9b32b47e891747d60488e7320"
format_stamp: "Formatted at 2026-09-23 17:34:33 on dist-test-slave-ppjl"
server_key: "f420d5b16ecf3e8ef3c096c7e81a2caf"
server_key_iv: "2a0fb2b976477e43c3534284916805ef"
server_key_version: "encryptionkey@0"
I20260923 17:34:33.091650 22343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:34:33.095405 22358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:33.096939 22343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:34:33.097119 22343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "c611c3b9b32b47e891747d60488e7320"
format_stamp: "Formatted at 2026-09-23 17:34:33 on dist-test-slave-ppjl"
server_key: "f420d5b16ecf3e8ef3c096c7e81a2caf"
server_key_iv: "2a0fb2b976477e43c3534284916805ef"
server_key_version: "encryptionkey@0"
I20260923 17:34:33.097332 22343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:33.123327 22343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:33.127861 22343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:33.128166 22343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:33.129196 22343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:34:33.130681 22343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:33.130752 22343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:33.130846 22343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:33.130906 22343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:33.162586 22343 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:34849
I20260923 17:34:33.162632 22470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:34849 every 8 connection(s)
I20260923 17:34:33.163878 22343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:34:33.172515 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 22343
I20260923 17:34:33.172722 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:34:33.173269 21441 external_mini_cluster.cc:1428] Setting key de0aff9b44e514a4d9eabcedc2300685
Sep 23 17:34:33 dist-test-slave-ppjl krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184873, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:33.185940 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.166481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:52023 (local address 127.20.240.126:35799)
0923 17:34:33.166724 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:33.166729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:33.168003 (+ 1274us) server_negotiation.cc:407] Connection header received
0923 17:34:33.170087 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:33.170094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:33.170200 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:33.170304 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:33.172214 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.173318 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.175434 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.175984 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.176115 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.179778 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:33.179785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:33.179790 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:33.179833 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:33.182614 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.183481 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.183486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.183488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.183629 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.184091 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.184095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.184097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.184413 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:33.184611 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:33.185348 (+ 737us) server_negotiation.cc:299] Negotiation successful
0923 17:34:33.185586 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260923 17:34:33.186930 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.166960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35799 (local address 127.20.240.67:52023)
0923 17:34:33.167780 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:33.167836 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:33.169727 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:33.170611 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:33.170622 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:33.171299 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:33.171965 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:33.171987 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.173528 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.173533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:33.175235 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:33.175258 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.176262 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.176270 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:33.176556 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.177234 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:33.177258 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:33.179533 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:33.182762 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:33.182769 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:33.182784 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:33.183339 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:33.183745 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:33.183749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:33.183751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:33.183937 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:33.184601 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:33.184609 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:33.184968 (+ 359us) client_negotiation.cc:769] Sending connection context
0923 17:34:33.185209 (+ 241us) client_negotiation.cc:240] Negotiation successful
0923 17:34:33.185759 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":360,"threads_started":1}
I20260923 17:34:33.188616 22471 heartbeater.cc:347] Connected to a master server at 127.20.240.126:35799
I20260923 17:34:33.188958 22471 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:33.189900 22471 heartbeater.cc:511] Master 127.20.240.126:35799 requested a full tablet report, sending...
I20260923 17:34:33.191440 22014 ts_manager.cc:194] Registered new tserver with Master: c611c3b9b32b47e891747d60488e7320 (127.20.240.67:34849)
I20260923 17:34:33.192340 22014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:52023
I20260923 17:34:33.202484 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:34:33.202922 22204 heartbeater.cc:503] Master 127.20.240.126:35799 was elected leader, sending a full tablet report...
I20260923 17:34:33.222477 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.209306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44310 (local address 127.20.240.126:35799)
0923 17:34:33.209536 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:33.209544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:33.209949 (+ 405us) server_negotiation.cc:407] Connection header received
0923 17:34:33.210097 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:33.210101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:33.210165 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:33.210245 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:33.211851 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.212649 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.213967 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.214385 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.214494 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.217090 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:33.217095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:33.217098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:33.217140 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:33.219724 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.220423 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.220427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.220429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.220568 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.220956 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.220960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.220962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.221341 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:33.221530 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:33.221950 (+ 420us) server_negotiation.cc:299] Negotiation successful
0923 17:34:33.222133 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 17:34:33.229605 22014 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:34:33.236771 22014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:34:33.265856 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.250929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.20.240.65:37997)
0923 17:34:33.251925 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:33.251937 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:33.256527 (+ 4590us) server_negotiation.cc:407] Connection header received
0923 17:34:33.257011 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:33.257020 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:33.257329 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:33.257487 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:33.258702 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.260392 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.262814 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.264205 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.264347 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.264621 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:33.265304 (+ 683us) server_negotiation.cc:299] Negotiation successful
0923 17:34:33.265512 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":703,"threads_started":1}
I20260923 17:34:33.265941 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.250926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37808 (local address 127.20.240.66:34083)
0923 17:34:33.252363 (+ 1437us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:33.252373 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:33.256504 (+ 4131us) server_negotiation.cc:407] Connection header received
0923 17:34:33.257048 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:33.257057 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:33.257310 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:33.257488 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:33.258671 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.260399 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.263465 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.264645 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.264773 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.264984 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:33.265544 (+ 560us) server_negotiation.cc:299] Negotiation successful
0923 17:34:33.265680 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1276,"thread_start_us":1147,"threads_started":1}
I20260923 17:34:33.266801 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.250585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:37997 (local address 127.0.0.1:35234)
0923 17:34:33.256366 (+ 5781us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:33.256388 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:33.256713 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:33.257808 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:33.257814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:33.257857 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:33.258468 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:33.258482 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.260566 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.260572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:33.262629 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:33.262643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.264777 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.264784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:33.264996 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.265069 (+ 73us) client_negotiation.cc:769] Sending connection context
0923 17:34:33.266139 (+ 1070us) client_negotiation.cc:240] Negotiation successful
0923 17:34:33.266368 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5614,"mutex_wait_us":8,"spinlock_wait_cycles":9856,"thread_start_us":6309,"threads_started":1}
I20260923 17:34:33.266894 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.251360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:34849 (local address 127.0.0.1:53706)
0923 17:34:33.257158 (+ 5798us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:33.257175 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:33.257375 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:33.258231 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:33.258236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:33.258257 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:33.258718 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:33.258727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.260394 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.260399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:33.263254 (+ 2855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:33.263270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.265336 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.265342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:33.266380 (+ 1038us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.266402 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 17:34:33.266479 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 17:34:33.266573 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5684,"mutex_wait_us":200,"thread_start_us":4396,"threads_started":1}
I20260923 17:34:33.267084 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.251674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.20.240.67:34849)
0923 17:34:33.255827 (+ 4153us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:33.255837 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:33.257276 (+ 1439us) server_negotiation.cc:407] Connection header received
0923 17:34:33.257566 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:33.257574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:33.257790 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:33.257942 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:33.258896 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.260208 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.263436 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.264824 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.264966 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.265193 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:33.266572 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0923 17:34:33.266755 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3948,"thread_start_us":3766,"threads_started":1}
I20260923 17:34:33.267482 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.250480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:34083 (local address 127.0.0.1:37808)
0923 17:34:33.256299 (+ 5819us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:33.256354 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:33.256714 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:33.257806 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:33.257814 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:33.257854 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:33.258480 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:33.258490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.260566 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.260572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:33.263262 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:33.263271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.265137 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.265143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:33.265319 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.265351 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:34:33.266185 (+ 834us) client_negotiation.cc:240] Negotiation successful
0923 17:34:33.267149 (+ 964us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5527,"mutex_wait_us":1055,"spinlock_wait_cycles":13696,"thread_start_us":6097,"threads_started":1}
I20260923 17:34:33.270874 22406 tablet_service.cc:1467] Processing CreateTablet for tablet 7be9bdd3db8a4fd0b861471cb33fdc70 (DEFAULT_TABLE table=test-table [id=787c07086d934f8fa5a0a2de77e507fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:33.270885 22273 tablet_service.cc:1467] Processing CreateTablet for tablet 7be9bdd3db8a4fd0b861471cb33fdc70 (DEFAULT_TABLE table=test-table [id=787c07086d934f8fa5a0a2de77e507fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:33.272245 22406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be9bdd3db8a4fd0b861471cb33fdc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:33.272308 22273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be9bdd3db8a4fd0b861471cb33fdc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:33.274127 22139 tablet_service.cc:1467] Processing CreateTablet for tablet 7be9bdd3db8a4fd0b861471cb33fdc70 (DEFAULT_TABLE table=test-table [id=787c07086d934f8fa5a0a2de77e507fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:33.276108 22139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be9bdd3db8a4fd0b861471cb33fdc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:33.284547 22489 tablet_bootstrap.cc:492] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320: Bootstrap starting.
I20260923 17:34:33.286979 22490 tablet_bootstrap.cc:492] T 7be9bdd3db8a4fd0b861471cb33fdc70 P 08969e2e21644610bfd47eb5acb19481: Bootstrap starting.
I20260923 17:34:33.289305 22489 tablet_bootstrap.cc:654] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:33.290649 22490 tablet_bootstrap.cc:654] T 7be9bdd3db8a4fd0b861471cb33fdc70 P 08969e2e21644610bfd47eb5acb19481: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:33.290642 22489 log.cc:824] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:33.291381 22491 tablet_bootstrap.cc:492] T 7be9bdd3db8a4fd0b861471cb33fdc70 P be36632deef7488e9d95aed20a78a3d7: Bootstrap starting.
I20260923 17:34:33.291967 22490 log.cc:824] T 7be9bdd3db8a4fd0b861471cb33fdc70 P 08969e2e21644610bfd47eb5acb19481: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:33.294854 22490 tablet_bootstrap.cc:492] T 7be9bdd3db8a4fd0b861471cb33fdc70 P 08969e2e21644610bfd47eb5acb19481: No bootstrap required, opened a new log
I20260923 17:34:33.295076 22491 tablet_bootstrap.cc:654] T 7be9bdd3db8a4fd0b861471cb33fdc70 P be36632deef7488e9d95aed20a78a3d7: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:33.295106 22490 ts_tablet_manager.cc:1397] T 7be9bdd3db8a4fd0b861471cb33fdc70 P 08969e2e21644610bfd47eb5acb19481: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 17:34:33.295264 22489 tablet_bootstrap.cc:492] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320: No bootstrap required, opened a new log
I20260923 17:34:33.295547 22489 ts_tablet_manager.cc:1397] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260923 17:34:33.296453 22491 log.cc:824] T 7be9bdd3db8a4fd0b861471cb33fdc70 P be36632deef7488e9d95aed20a78a3d7: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:33.303376 22491 tablet_bootstrap.cc:492] T 7be9bdd3db8a4fd0b861471cb33fdc70 P be36632deef7488e9d95aed20a78a3d7: No bootstrap required, opened a new log
I20260923 17:34:33.303723 22491 ts_tablet_manager.cc:1397] T 7be9bdd3db8a4fd0b861471cb33fdc70 P be36632deef7488e9d95aed20a78a3d7: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260923 17:34:33.303740 22490 raft_consensus.cc:348] T 7be9bdd3db8a4fd0b861471cb33fdc70 P 08969e2e21644610bfd47eb5acb19481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08969e2e21644610bfd47eb5acb19481" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34083 } } peers { permanent_uuid: "be36632deef7488e9d95aed20a78a3d7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37997 } } peers { permanent_uuid: "c611c3b9b32b47e891747d60488e7320" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34849 } }
I20260923 17:34:33.304167 22490 raft_consensus.cc:374] T 7be9bdd3db8a4fd0b861471cb33fdc70 P 08969e2e21644610bfd47eb5acb19481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:33.304258 22490 raft_consensus.cc:729] T 7be9bdd3db8a4fd0b861471cb33fdc70 P 08969e2e21644610bfd47eb5acb19481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08969e2e21644610bfd47eb5acb19481, State: Initialized, Role: FOLLOWER
I20260923 17:34:33.304944 22490 consensus_queue.cc:260] T 7be9bdd3db8a4fd0b861471cb33fdc70 P 08969e2e21644610bfd47eb5acb19481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08969e2e21644610bfd47eb5acb19481" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34083 } } peers { permanent_uuid: "be36632deef7488e9d95aed20a78a3d7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37997 } } peers { permanent_uuid: "c611c3b9b32b47e891747d60488e7320" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34849 } }
I20260923 17:34:33.305785 22489 raft_consensus.cc:348] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08969e2e21644610bfd47eb5acb19481" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34083 } } peers { permanent_uuid: "be36632deef7488e9d95aed20a78a3d7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37997 } } peers { permanent_uuid: "c611c3b9b32b47e891747d60488e7320" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34849 } }
I20260923 17:34:33.306300 22489 raft_consensus.cc:374] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:33.306420 22489 raft_consensus.cc:729] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c611c3b9b32b47e891747d60488e7320, State: Initialized, Role: FOLLOWER
I20260923 17:34:33.307222 22489 consensus_queue.cc:260] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08969e2e21644610bfd47eb5acb19481" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34083 } } peers { permanent_uuid: "be36632deef7488e9d95aed20a78a3d7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37997 } } peers { permanent_uuid: "c611c3b9b32b47e891747d60488e7320" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34849 } }
I20260923 17:34:33.312127 22471 heartbeater.cc:503] Master 127.20.240.126:35799 was elected leader, sending a full tablet report...
I20260923 17:34:33.312907 22489 ts_tablet_manager.cc:1428] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260923 17:34:33.313396 22490 ts_tablet_manager.cc:1428] T 7be9bdd3db8a4fd0b861471cb33fdc70 P 08969e2e21644610bfd47eb5acb19481: Time spent starting tablet: real 0.018s user 0.009s sys 0.007s
I20260923 17:34:33.313781 22338 heartbeater.cc:503] Master 127.20.240.126:35799 was elected leader, sending a full tablet report...
I20260923 17:34:33.314692 22491 raft_consensus.cc:348] T 7be9bdd3db8a4fd0b861471cb33fdc70 P be36632deef7488e9d95aed20a78a3d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08969e2e21644610bfd47eb5acb19481" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34083 } } peers { permanent_uuid: "be36632deef7488e9d95aed20a78a3d7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37997 } } peers { permanent_uuid: "c611c3b9b32b47e891747d60488e7320" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34849 } }
I20260923 17:34:33.315228 22491 raft_consensus.cc:374] T 7be9bdd3db8a4fd0b861471cb33fdc70 P be36632deef7488e9d95aed20a78a3d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:33.315346 22491 raft_consensus.cc:729] T 7be9bdd3db8a4fd0b861471cb33fdc70 P be36632deef7488e9d95aed20a78a3d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be36632deef7488e9d95aed20a78a3d7, State: Initialized, Role: FOLLOWER
I20260923 17:34:33.316036 22491 consensus_queue.cc:260] T 7be9bdd3db8a4fd0b861471cb33fdc70 P be36632deef7488e9d95aed20a78a3d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08969e2e21644610bfd47eb5acb19481" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34083 } } peers { permanent_uuid: "be36632deef7488e9d95aed20a78a3d7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37997 } } peers { permanent_uuid: "c611c3b9b32b47e891747d60488e7320" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34849 } }
I20260923 17:34:33.320477 22491 ts_tablet_manager.cc:1428] T 7be9bdd3db8a4fd0b861471cb33fdc70 P be36632deef7488e9d95aed20a78a3d7: Time spent starting tablet: real 0.017s user 0.005s sys 0.011s
W20260923 17:34:33.406206 22339 tablet.cc:2369] T 7be9bdd3db8a4fd0b861471cb33fdc70 P 08969e2e21644610bfd47eb5acb19481: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:34:33.415506 22472 tablet.cc:2369] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:34:33.422986 22205 tablet.cc:2369] T 7be9bdd3db8a4fd0b861471cb33fdc70 P be36632deef7488e9d95aed20a78a3d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:34:33.449898 22496 raft_consensus.cc:482] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:34:33.450261 22496 raft_consensus.cc:504] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08969e2e21644610bfd47eb5acb19481" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34083 } } peers { permanent_uuid: "be36632deef7488e9d95aed20a78a3d7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37997 } } peers { permanent_uuid: "c611c3b9b32b47e891747d60488e7320" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34849 } }
I20260923 17:34:33.452050 22496 leader_election.cc:288] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08969e2e21644610bfd47eb5acb19481 (127.20.240.66:34083), be36632deef7488e9d95aed20a78a3d7 (127.20.240.65:37997)
I20260923 17:34:33.458807 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.452583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:59953 (local address 127.20.240.66:34083)
0923 17:34:33.452903 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:33.452909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:33.452930 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:34:33.453114 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:33.453118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:33.453179 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:33.453257 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:33.453829 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.454799 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.456753 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.457862 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.457987 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.458044 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:33.458494 (+ 450us) server_negotiation.cc:299] Negotiation successful
0923 17:34:33.458604 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260923 17:34:33.458896 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.452398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:34083 (local address 127.20.240.67:59953)
0923 17:34:33.452789 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:33.452811 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:33.452960 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:33.453431 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:33.453434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:33.453461 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:33.453667 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:33.453679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.454970 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.454976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:33.456578 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:33.456592 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.458046 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.458053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:33.458284 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.458324 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 17:34:33.458439 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 17:34:33.458593 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260923 17:34:33.460035 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.452636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:39453 (local address 127.20.240.65:37997)
0923 17:34:33.452873 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:33.452879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:33.453501 (+ 622us) server_negotiation.cc:407] Connection header received
0923 17:34:33.453653 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:33.453658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:33.453733 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:33.453876 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:33.454541 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.455527 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.457677 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.458942 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.459107 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.459215 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:33.459656 (+ 441us) server_negotiation.cc:299] Negotiation successful
0923 17:34:33.459790 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:34:33.460068 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.452532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:37997 (local address 127.20.240.67:39453)
0923 17:34:33.453389 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:33.453408 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:33.453527 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:33.453915 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:33.453919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:33.453945 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:33.454309 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:33.454321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.455685 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.455689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:33.457475 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:33.457491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.459348 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.459352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:33.459476 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.459500 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:34:33.459580 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 17:34:33.459739 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":534,"threads_started":1}
I20260923 17:34:33.459982 22293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be9bdd3db8a4fd0b861471cb33fdc70" candidate_uuid: "c611c3b9b32b47e891747d60488e7320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08969e2e21644610bfd47eb5acb19481" is_pre_election: true
I20260923 17:34:33.460592 22293 raft_consensus.cc:2454] T 7be9bdd3db8a4fd0b861471cb33fdc70 P 08969e2e21644610bfd47eb5acb19481 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c611c3b9b32b47e891747d60488e7320 in term 0.
I20260923 17:34:33.461403 22359 leader_election.cc:302] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08969e2e21644610bfd47eb5acb19481, c611c3b9b32b47e891747d60488e7320; no voters:
I20260923 17:34:33.461308 22159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be9bdd3db8a4fd0b861471cb33fdc70" candidate_uuid: "c611c3b9b32b47e891747d60488e7320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be36632deef7488e9d95aed20a78a3d7" is_pre_election: true
I20260923 17:34:33.461804 22496 raft_consensus.cc:2781] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:34:33.461882 22159 raft_consensus.cc:2454] T 7be9bdd3db8a4fd0b861471cb33fdc70 P be36632deef7488e9d95aed20a78a3d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c611c3b9b32b47e891747d60488e7320 in term 0.
I20260923 17:34:33.461980 22496 raft_consensus.cc:482] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:34:33.462082 22496 raft_consensus.cc:3037] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:33.464551 22496 raft_consensus.cc:504] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08969e2e21644610bfd47eb5acb19481" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34083 } } peers { permanent_uuid: "be36632deef7488e9d95aed20a78a3d7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37997 } } peers { permanent_uuid: "c611c3b9b32b47e891747d60488e7320" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34849 } }
I20260923 17:34:33.465157 22496 leader_election.cc:288] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [CANDIDATE]: Term 1 election: Requested vote from peers 08969e2e21644610bfd47eb5acb19481 (127.20.240.66:34083), be36632deef7488e9d95aed20a78a3d7 (127.20.240.65:37997)
I20260923 17:34:33.465796 22159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be9bdd3db8a4fd0b861471cb33fdc70" candidate_uuid: "c611c3b9b32b47e891747d60488e7320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be36632deef7488e9d95aed20a78a3d7"
I20260923 17:34:33.465793 22293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be9bdd3db8a4fd0b861471cb33fdc70" candidate_uuid: "c611c3b9b32b47e891747d60488e7320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08969e2e21644610bfd47eb5acb19481"
I20260923 17:34:33.466028 22293 raft_consensus.cc:3037] T 7be9bdd3db8a4fd0b861471cb33fdc70 P 08969e2e21644610bfd47eb5acb19481 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:33.466028 22159 raft_consensus.cc:3037] T 7be9bdd3db8a4fd0b861471cb33fdc70 P be36632deef7488e9d95aed20a78a3d7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:33.469329 22293 raft_consensus.cc:2454] T 7be9bdd3db8a4fd0b861471cb33fdc70 P 08969e2e21644610bfd47eb5acb19481 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c611c3b9b32b47e891747d60488e7320 in term 1.
I20260923 17:34:33.469329 22159 raft_consensus.cc:2454] T 7be9bdd3db8a4fd0b861471cb33fdc70 P be36632deef7488e9d95aed20a78a3d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c611c3b9b32b47e891747d60488e7320 in term 1.
I20260923 17:34:33.469897 22360 leader_election.cc:302] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be36632deef7488e9d95aed20a78a3d7, c611c3b9b32b47e891747d60488e7320; no voters:
I20260923 17:34:33.470263 22496 raft_consensus.cc:2781] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:33.470743 22496 raft_consensus.cc:686] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [term 1 LEADER]: Becoming Leader. State: Replica: c611c3b9b32b47e891747d60488e7320, State: Running, Role: LEADER
I20260923 17:34:33.471263 22496 consensus_queue.cc:237] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08969e2e21644610bfd47eb5acb19481" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34083 } } peers { permanent_uuid: "be36632deef7488e9d95aed20a78a3d7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37997 } } peers { permanent_uuid: "c611c3b9b32b47e891747d60488e7320" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34849 } }
I20260923 17:34:33.476518 22015 catalog_manager.cc:5495] T 7be9bdd3db8a4fd0b861471cb33fdc70 P c611c3b9b32b47e891747d60488e7320 reported cstate change: term changed from 0 to 1, leader changed from <none> to c611c3b9b32b47e891747d60488e7320 (127.20.240.67). New cstate: current_term: 1 leader_uuid: "c611c3b9b32b47e891747d60488e7320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08969e2e21644610bfd47eb5acb19481" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be36632deef7488e9d95aed20a78a3d7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c611c3b9b32b47e891747d60488e7320" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34849 } health_report { overall_health: HEALTHY } } }
Sep 23 17:34:33 dist-test-slave-ppjl krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:34:33.513842 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.498252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.20.240.65:37997)
0923 17:34:33.498433 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:33.498439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:33.498939 (+ 500us) server_negotiation.cc:407] Connection header received
0923 17:34:33.499099 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:33.499103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:33.499213 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:33.499299 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:33.500810 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.501569 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.502814 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.503310 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.503424 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.507862 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:33.507879 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:33.507898 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:33.507937 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:33.510711 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.511673 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.511682 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.511684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.511848 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.512315 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.512319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.512322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.512714 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:33.512960 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:33.513308 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 17:34:33.513502 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 17:34:33 dist-test-slave-ppjl krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184873, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:34:33 dist-test-slave-ppjl krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184873, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:34:33.560266 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.545442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35258 (local address 127.20.240.65:37997)
0923 17:34:33.545655 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:33.545661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:33.545970 (+ 309us) server_negotiation.cc:407] Connection header received
0923 17:34:33.546128 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:33.546132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:33.546197 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:33.546281 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:33.547685 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.548425 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.549737 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.550153 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.550266 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.554458 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:33.554465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:33.554469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:33.554514 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:33.557487 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.558237 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.558242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.558244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.558396 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.558837 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.558841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.558843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.559130 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:33.559418 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:33.559705 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 17:34:33.559928 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Sep 23 17:34:33 dist-test-slave-ppjl krb5kdc[21970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:34:33 dist-test-slave-ppjl krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:34:33.603863 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.589116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.20.240.65:37997)
0923 17:34:33.589443 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:33.589450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:33.589718 (+ 268us) server_negotiation.cc:407] Connection header received
0923 17:34:33.589910 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:33.589915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:33.589999 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:33.590084 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:33.591602 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.592294 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.593529 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.593960 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.594075 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.598426 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:33.598440 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:33.598443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:33.598485 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:33.601111 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.601841 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.601846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.601848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.601979 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.602359 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.602363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.602365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.602683 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:33.602919 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:33.603276 (+ 357us) server_negotiation.cc:299] Negotiation successful
0923 17:34:33.603528 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 17:34:33.605931 22119 tablet_service.cc:2908] Rejecting scan request for tablet 7be9bdd3db8a4fd0b861471cb33fdc70: Uninitialized: clean time has not yet been initialized
I20260923 17:34:33.609082 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 22076
I20260923 17:34:33.622546 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.610166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.20.240.65:37997)
0923 17:34:33.610384 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:33.610392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:33.610762 (+ 370us) server_negotiation.cc:407] Connection header received
0923 17:34:33.610918 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:33.610922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:33.611007 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:33.611090 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:33.612554 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.613253 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.614502 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.614916 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.615022 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.617366 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:33.617371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:33.617375 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:33.617467 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:33.620026 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.620753 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.620758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.620760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.620906 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.621206 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.621209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.621211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.621545 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:33.621780 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:33.622015 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 17:34:33.622201 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:34:33.623234 22199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:34:33.713212 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 22076
I20260923 17:34:33.726817 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 22210
Sep 23 17:34:33 dist-test-slave-ppjl krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:34:33.741640 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.727568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.20.240.66:34083)
0923 17:34:33.727765 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:33.727770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:33.727815 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:34:33.727899 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:33.727902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:33.728000 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:33.728082 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:33.729334 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.730117 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.731616 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.732058 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.732163 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.736071 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:33.736088 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:33.736106 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:33.736144 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:33.738908 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.739751 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.739758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.739761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.739891 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.740245 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.740248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.740250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.740634 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:33.740863 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:33.741133 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 17:34:33.741303 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 17:34:33.742461 22333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:34:33.832520 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 22210
I20260923 17:34:33.846272 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 22343
Sep 23 17:34:33 dist-test-slave-ppjl krb5kdc[21970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:34:33.863940 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:33.847115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53712 (local address 127.20.240.67:34849)
0923 17:34:33.847784 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:33.847789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:33.847806 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:34:33.847861 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:33.847864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:33.847936 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:33.848030 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:33.849499 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.850254 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.851769 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:33.852220 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:33.852339 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:33.856920 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:33.856941 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:33.856959 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:33.857043 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:33.860445 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.861476 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.861485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.861487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.861666 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:33.862123 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:33.862128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:33.862130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:33.862644 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:33.862890 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:33.863291 (+ 401us) server_negotiation.cc:299] Negotiation successful
0923 17:34:33.863558 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":282,"threads_started":1}
I20260923 17:34:33.864722 22466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:34:33.955946 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 22343
I20260923 17:34:33.969319 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 21983
I20260923 17:34:33.970544 22045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:34:34.079496 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 21983
2026-09-23T17:34:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4161 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:34:34 dist-test-slave-ppjl krb5kdc[22519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:34:34 dist-test-slave-ppjl krb5kdc[22519](info): set up 2 sockets
Sep 23 17:34:34 dist-test-slave-ppjl krb5kdc[22519](info): commencing operation
krb5kdc: starting...
W20260923 17:34:35.087966 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:34:35 dist-test-slave-ppjl krb5kdc[22519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:34:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:34:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:34:35.276099 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:38733
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:38733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:35.592463 22532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:35.592789 22532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:35.592934 22532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:35.600229 22532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:34:35.600353 22532 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 17:34:35.600423 22532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:35.600508 22532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:35.600577 22532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:34:35.600658 22532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:34:35.610699 22532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:38733
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:38733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:35.612102 22532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:35.613443 22532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:35.624869 22537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:35.624945 22538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:35.625001 22540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:35.626425 22532 server_base.cc:1034] running on GCE node
I20260923 17:34:35.627282 22532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:35.628751 22532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:35.629993 22532 hybrid_clock.cc:648] HybridClock initialized: now 1790184875629929 us; error 44 us; skew 500 ppm
I20260923 17:34:35.630319 22532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:35 dist-test-slave-ppjl krb5kdc[22519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184875, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:35.633474 22532 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:34:35.635038 22532 webserver.cc:460] Webserver started at http://127.20.240.126:44653/ using document root <none> and password file <none>
I20260923 17:34:35.635795 22532 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:35.635908 22532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:35.636230 22532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:35.639335 22532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "fb5e648d26874de98cae4a66cf55d66d"
format_stamp: "Formatted at 2026-09-23 17:34:35 on dist-test-slave-ppjl"
server_key: "196d30c57060420906d0d03028487b17"
server_key_iv: "7a261d7b558fb97e97b141895b98ebb2"
server_key_version: "encryptionkey@0"
I20260923 17:34:35.640157 22532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "fb5e648d26874de98cae4a66cf55d66d"
format_stamp: "Formatted at 2026-09-23 17:34:35 on dist-test-slave-ppjl"
server_key: "196d30c57060420906d0d03028487b17"
server_key_iv: "7a261d7b558fb97e97b141895b98ebb2"
server_key_version: "encryptionkey@0"
I20260923 17:34:35.645624 22532 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 17:34:35.649174 22547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:35.650638 22532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:34:35.650802 22532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "fb5e648d26874de98cae4a66cf55d66d"
format_stamp: "Formatted at 2026-09-23 17:34:35 on dist-test-slave-ppjl"
server_key: "196d30c57060420906d0d03028487b17"
server_key_iv: "7a261d7b558fb97e97b141895b98ebb2"
server_key_version: "encryptionkey@0"
I20260923 17:34:35.651026 22532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:35.704190 22532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:35.708647 22532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:35.708982 22532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:35.732005 22532 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:38733
I20260923 17:34:35.732048 22608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:38733 every 8 connection(s)
I20260923 17:34:35.733490 22532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:34:35.734225 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 22532
I20260923 17:34:35.734439 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:34:35.735244 21441 external_mini_cluster.cc:1428] Setting key 33471aef5a4a68232cfafa1a0262513d
I20260923 17:34:35.738386 22609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:34:35 dist-test-slave-ppjl krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:35.752564 22609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d: Bootstrap starting.
I20260923 17:34:35.757059 22609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:35.758342 22609 log.cc:824] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:35.761104 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:35.737392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37524 (local address 127.20.240.126:38733)
0923 17:34:35.738150 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:35.738162 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:35.738202 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:34:35.739949 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:35.739973 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:35.740377 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:35.740724 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:35.742685 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:35.744215 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:35.746105 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:35.746809 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:35.746950 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:35.752207 (+ 5257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:35.752221 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:35.752244 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:35.752302 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:35.756492 (+ 4190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:35.757249 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:35.757258 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:35.757267 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:35.757459 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:35.758015 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:35.758019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:35.758021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:35.758678 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:35.758956 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:35.759631 (+ 675us) server_negotiation.cc:299] Negotiation successful
0923 17:34:35.760080 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":350,"threads_started":1}
I20260923 17:34:35.761577 22609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d: No bootstrap required, opened a new log
I20260923 17:34:35.769668 22609 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5e648d26874de98cae4a66cf55d66d" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 38733 } }
I20260923 17:34:35.770104 22609 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:35.770208 22609 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5e648d26874de98cae4a66cf55d66d, State: Initialized, Role: FOLLOWER
I20260923 17:34:35.770814 22609 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5e648d26874de98cae4a66cf55d66d" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 38733 } }
I20260923 17:34:35.771050 22609 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:34:35.771202 22609 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:34:35.771374 22609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:35.773856 22609 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5e648d26874de98cae4a66cf55d66d" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 38733 } }
I20260923 17:34:35.774367 22609 leader_election.cc:302] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb5e648d26874de98cae4a66cf55d66d; no voters:
I20260923 17:34:35.774942 22609 leader_election.cc:288] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:34:35.775072 22614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:35.775841 22614 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [term 1 LEADER]: Becoming Leader. State: Replica: fb5e648d26874de98cae4a66cf55d66d, State: Running, Role: LEADER
I20260923 17:34:35.776371 22614 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5e648d26874de98cae4a66cf55d66d" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 38733 } }
I20260923 17:34:35.777053 22609 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:34:35.779111 22616 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb5e648d26874de98cae4a66cf55d66d. Latest consensus state: current_term: 1 leader_uuid: "fb5e648d26874de98cae4a66cf55d66d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5e648d26874de98cae4a66cf55d66d" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 38733 } } }
I20260923 17:34:35.779529 22616 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:35.779080 22615 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb5e648d26874de98cae4a66cf55d66d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb5e648d26874de98cae4a66cf55d66d" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 38733 } } }
I20260923 17:34:35.779750 22615 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:35.780567 22620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:34:35.786792 22620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:34:35.798406 22620 catalog_manager.cc:1347] Generated new cluster ID: a3aa13e424464b25a06509af766dc9fd
I20260923 17:34:35.798533 22620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:34:35.813980 22620 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:34:35.815444 22620 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:34:35.826912 22620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d: Generated new TSK 0
I20260923 17:34:35.827895 22620 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:34:35.875448 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:38733
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:36.205086 22635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:36.205379 22635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:36.205500 22635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:36.212810 22635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:36.212924 22635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:36.213054 22635 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:34:36.213164 22635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:34:36.223670 22635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.240.126:38733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:36.225037 22635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:36.226481 22635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:36.240095 22641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:36.240135 22640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:36.240176 22643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:36.241220 22635 server_base.cc:1034] running on GCE node
I20260923 17:34:36.241890 22635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:36.242914 22635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:36.244156 22635 hybrid_clock.cc:648] HybridClock initialized: now 1790184876244070 us; error 58 us; skew 500 ppm
I20260923 17:34:36.244557 22635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:36 dist-test-slave-ppjl krb5kdc[22519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184876, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:36.248082 22635 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:34:36.250001 22635 webserver.cc:460] Webserver started at http://127.20.240.65:44857/ using document root <none> and password file <none>
I20260923 17:34:36.250762 22635 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:36.250869 22635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:36.251236 22635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:36.254360 22635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb"
format_stamp: "Formatted at 2026-09-23 17:34:36 on dist-test-slave-ppjl"
server_key: "fc91a1cbec6772caf7fcfc1513174ce6"
server_key_iv: "c3851fa27a561f84510a6711463f73e5"
server_key_version: "encryptionkey@0"
I20260923 17:34:36.255285 22635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb"
format_stamp: "Formatted at 2026-09-23 17:34:36 on dist-test-slave-ppjl"
server_key: "fc91a1cbec6772caf7fcfc1513174ce6"
server_key_iv: "c3851fa27a561f84510a6711463f73e5"
server_key_version: "encryptionkey@0"
I20260923 17:34:36.261085 22635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 17:34:36.264739 22650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:36.266184 22635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:34:36.266350 22635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb"
format_stamp: "Formatted at 2026-09-23 17:34:36 on dist-test-slave-ppjl"
server_key: "fc91a1cbec6772caf7fcfc1513174ce6"
server_key_iv: "c3851fa27a561f84510a6711463f73e5"
server_key_version: "encryptionkey@0"
I20260923 17:34:36.266575 22635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:36.299291 22635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:36.304039 22635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:36.304303 22635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:36.307001 22635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:36.307089 22635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:36.307240 22635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:36.307317 22635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:34:36 dist-test-slave-ppjl krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184876, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:36.331472 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:36.307942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:38675 (local address 127.20.240.126:38733)
0923 17:34:36.308345 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:36.308350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:36.309590 (+ 1240us) server_negotiation.cc:407] Connection header received
0923 17:34:36.311602 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:36.311608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:36.311697 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:36.311824 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:36.314714 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.315647 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.318591 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.319312 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.319476 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:36.324751 (+ 5275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:36.324758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:36.324763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:36.324808 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:36.328420 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:36.329174 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:36.329179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:36.329181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:36.329300 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:36.329734 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:36.329737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:36.329740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:36.330018 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:36.330141 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:36.330848 (+ 707us) server_negotiation.cc:299] Negotiation successful
0923 17:34:36.331076 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":202,"threads_started":1}
I20260923 17:34:36.332917 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:36.308628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:38733 (local address 127.20.240.65:38675)
0923 17:34:36.309398 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:36.309435 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:36.311299 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:36.312304 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:36.312322 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:36.313275 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:36.314449 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:36.314486 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.315860 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.315866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:36.318341 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:36.318363 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.319808 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.319821 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:36.320487 (+ 666us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:36.321580 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:36.321617 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:36.324488 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:36.328586 (+ 4098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:36.328593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:36.328608 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:36.329036 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:36.329417 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:36.329421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:36.329423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:36.329614 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:36.330157 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:36.330165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:36.330506 (+ 341us) client_negotiation.cc:769] Sending connection context
0923 17:34:36.330743 (+ 237us) client_negotiation.cc:240] Negotiation successful
0923 17:34:36.331316 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":332,"threads_started":1}
I20260923 17:34:36.351291 22635 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:39589
I20260923 17:34:36.351382 22770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:39589 every 8 connection(s)
I20260923 17:34:36.352810 22635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:34:36.353348 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 22635
I20260923 17:34:36.353601 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:34:36.354440 21441 external_mini_cluster.cc:1428] Setting key d6bb8be1c64d58e0ddd6d63f393d66cc
I20260923 17:34:36.356209 22771 heartbeater.cc:347] Connected to a master server at 127.20.240.126:38733
I20260923 17:34:36.356686 22771 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:36.357513 22771 heartbeater.cc:511] Master 127.20.240.126:38733 requested a full tablet report, sending...
I20260923 17:34:36.359973 22564 ts_manager.cc:194] Registered new tserver with Master: 9908ba2c3f2f4b88a520c0ea8fa8edcb (127.20.240.65:39589)
I20260923 17:34:36.360975 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:36.346292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:42545 (local address 127.20.240.126:38733)
0923 17:34:36.346522 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:36.346528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:36.347116 (+ 588us) server_negotiation.cc:407] Connection header received
0923 17:34:36.347330 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:36.347336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:36.347432 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:36.347553 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:36.348931 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.349995 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.351855 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.352464 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.352591 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:36.354503 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:36.354510 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:36.354514 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:36.354556 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:36.358298 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:36.359026 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:36.359032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:36.359035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:36.359204 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:36.359545 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:36.359550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:36.359554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:36.359836 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:36.359977 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:36.360394 (+ 417us) server_negotiation.cc:299] Negotiation successful
0923 17:34:36.360605 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:34:36.361022 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:36.346390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:38733 (local address 127.20.240.65:42545)
0923 17:34:36.346995 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:36.347015 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:36.347178 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:36.347608 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:36.347613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:36.348306 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:36.348727 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:36.348745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.350163 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.350172 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:36.351694 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:36.351711 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.352658 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.352664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:36.352790 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:36.353392 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:36.353410 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:36.354301 (+ 891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:36.358433 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:36.358438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:36.358442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:36.358905 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:36.359299 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:36.359304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:36.359308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:36.359431 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:36.359956 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:36.359961 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:36.360133 (+ 172us) client_negotiation.cc:769] Sending connection context
0923 17:34:36.360255 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 17:34:36.360590 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":325,"threads_started":1}
I20260923 17:34:36.362241 22564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:38675
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:34:36.395785 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:38733
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:36.720727 22775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:36.721123 22775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:36.721240 22775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:36.728965 22775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:36.729113 22775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:36.729206 22775 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:34:36.729303 22775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:34:36.740029 22775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.240.126:38733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:36.741516 22775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:36.743045 22775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:36.757303 22781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:36.757262 22783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:36.757586 22780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:36.758546 22775 server_base.cc:1034] running on GCE node
I20260923 17:34:36.759289 22775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:36.760353 22775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:36.761627 22775 hybrid_clock.cc:648] HybridClock initialized: now 1790184876761539 us; error 52 us; skew 500 ppm
I20260923 17:34:36.761966 22775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:36 dist-test-slave-ppjl krb5kdc[22519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184876, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:36.770293 22775 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:34:36.772022 22775 webserver.cc:460] Webserver started at http://127.20.240.66:42113/ using document root <none> and password file <none>
I20260923 17:34:36.772755 22775 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:36.772851 22775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:36.773138 22775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:36.776255 22775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "2979bbaf9f264d79a6488d364451210e"
format_stamp: "Formatted at 2026-09-23 17:34:36 on dist-test-slave-ppjl"
server_key: "fc8dc52b37b229000af64fe7b70b3e6c"
server_key_iv: "c1653bd06ab43374740c298943081f70"
server_key_version: "encryptionkey@0"
I20260923 17:34:36.777127 22775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "2979bbaf9f264d79a6488d364451210e"
format_stamp: "Formatted at 2026-09-23 17:34:36 on dist-test-slave-ppjl"
server_key: "fc8dc52b37b229000af64fe7b70b3e6c"
server_key_iv: "c1653bd06ab43374740c298943081f70"
server_key_version: "encryptionkey@0"
I20260923 17:34:36.782883 22775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:34:36.786753 22790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:36.788434 22775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:34:36.788586 22775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "2979bbaf9f264d79a6488d364451210e"
format_stamp: "Formatted at 2026-09-23 17:34:36 on dist-test-slave-ppjl"
server_key: "fc8dc52b37b229000af64fe7b70b3e6c"
server_key_iv: "c1653bd06ab43374740c298943081f70"
server_key_version: "encryptionkey@0"
I20260923 17:34:36.788793 22775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:36.806213 22775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:36.810534 22775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:36.810765 22775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:36.813668 22775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:36.813778 22775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:36.813901 22775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:36.813969 22775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:34:36 dist-test-slave-ppjl krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184876, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:36.837558 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:36.814453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:37465 (local address 127.20.240.126:38733)
0923 17:34:36.814662 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:36.814669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:36.816130 (+ 1461us) server_negotiation.cc:407] Connection header received
0923 17:34:36.818258 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:36.818265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:36.818353 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:36.818442 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:36.820887 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.821855 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.824205 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.824907 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.825055 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:36.830334 (+ 5279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:36.830342 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:36.830349 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:36.830402 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:36.833768 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:36.834877 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:36.834883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:36.834885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:36.835037 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:36.835675 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:36.835679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:36.835682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:36.836005 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:36.836137 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:36.837110 (+ 973us) server_negotiation.cc:299] Negotiation successful
0923 17:34:36.837302 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 17:34:36.839210 22799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:36.815050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:38733 (local address 127.20.240.66:37465)
0923 17:34:36.815939 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:36.815989 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:36.817932 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:36.818772 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:36.818784 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:36.819718 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:36.820669 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:36.820695 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.822018 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.822024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:36.824021 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:36.824041 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.825215 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.825222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:36.825565 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:36.826446 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:36.826494 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:36.829981 (+ 3487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:36.833946 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:36.833958 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:36.833981 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:36.834681 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:36.835195 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:36.835201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:36.835205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:36.835500 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:36.836131 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:36.836141 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:36.836669 (+ 528us) client_negotiation.cc:769] Sending connection context
0923 17:34:36.836995 (+ 326us) client_negotiation.cc:240] Negotiation successful
0923 17:34:36.837608 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":352,"threads_started":1}
I20260923 17:34:36.858994 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:36.846294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:38733 (local address 127.20.240.66:33877)
0923 17:34:36.846879 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:36.846896 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:36.847055 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:36.847440 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:36.847444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:36.847891 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:36.848171 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:36.848181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.849214 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.849219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:36.850240 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:36.850254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.851078 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.851084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:36.851301 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:36.852034 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:36.852052 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:36.852798 (+ 746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:36.856236 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:36.856242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:36.856247 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:36.856720 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:36.857097 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:36.857102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:36.857106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:36.857225 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:36.857810 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:36.857815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:36.858039 (+ 224us) client_negotiation.cc:769] Sending connection context
0923 17:34:36.858188 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 17:34:36.858531 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":301,"threads_started":1}
I20260923 17:34:36.859768 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:36.846125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:33877 (local address 127.20.240.126:38733)
0923 17:34:36.846305 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:36.846311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:36.847015 (+ 704us) server_negotiation.cc:407] Connection header received
0923 17:34:36.847214 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:36.847220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:36.847300 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:36.847388 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:36.848325 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.849072 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.850401 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:36.850885 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:36.850998 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:36.853026 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:36.853033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:36.853038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:36.853080 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:36.856079 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:36.856845 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:36.856851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:36.856854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:36.856983 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:36.857333 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:36.857344 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:36.857346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:36.857687 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:36.857822 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:36.859354 (+ 1532us) server_negotiation.cc:299] Negotiation successful
0923 17:34:36.859555 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:34:36.870198 22775 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:36083
I20260923 17:34:36.870242 22909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:36083 every 8 connection(s)
I20260923 17:34:36.871579 22775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:34:36.873953 22910 heartbeater.cc:347] Connected to a master server at 127.20.240.126:38733
I20260923 17:34:36.874229 22910 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:36.874320 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 22775
I20260923 17:34:36.874615 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:34:36.875082 22910 heartbeater.cc:511] Master 127.20.240.126:38733 requested a full tablet report, sending...
I20260923 17:34:36.875150 21441 external_mini_cluster.cc:1428] Setting key d6a7ef011d98032a20dc65cd9d211446
I20260923 17:34:36.876662 22563 ts_manager.cc:194] Registered new tserver with Master: 2979bbaf9f264d79a6488d364451210e (127.20.240.66:36083)
I20260923 17:34:36.877599 22563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:37465
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:34:36.921252 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:38733
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:37.242282 22914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:37.242609 22914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:37.242738 22914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:37.250288 22914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:37.250427 22914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:37.250550 22914 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:34:37.250654 22914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:34:37.260769 22914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.240.126:38733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:37.262104 22914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:37.263509 22914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:37.276340 22920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:37.276429 22922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:37.276366 22919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:37.277547 22914 server_base.cc:1034] running on GCE node
I20260923 17:34:37.278214 22914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:37.279260 22914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:37.280491 22914 hybrid_clock.cc:648] HybridClock initialized: now 1790184877280409 us; error 55 us; skew 500 ppm
I20260923 17:34:37.280795 22914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:37 dist-test-slave-ppjl krb5kdc[22519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:37.283931 22914 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:34:37.285410 22914 webserver.cc:460] Webserver started at http://127.20.240.67:36731/ using document root <none> and password file <none>
I20260923 17:34:37.286128 22914 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:37.286224 22914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:37.286532 22914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:37.289464 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "f19426ade30947f081be4568f1ab29e9"
format_stamp: "Formatted at 2026-09-23 17:34:37 on dist-test-slave-ppjl"
server_key: "42a0429606754917d221a0a69123dce9"
server_key_iv: "364a5a70a47c5f06e98f27415372df11"
server_key_version: "encryptionkey@0"
I20260923 17:34:37.290237 22914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "f19426ade30947f081be4568f1ab29e9"
format_stamp: "Formatted at 2026-09-23 17:34:37 on dist-test-slave-ppjl"
server_key: "42a0429606754917d221a0a69123dce9"
server_key_iv: "364a5a70a47c5f06e98f27415372df11"
server_key_version: "encryptionkey@0"
I20260923 17:34:37.295540 22914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260923 17:34:37.299244 22929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:37.300732 22914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:34:37.300925 22914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "f19426ade30947f081be4568f1ab29e9"
format_stamp: "Formatted at 2026-09-23 17:34:37 on dist-test-slave-ppjl"
server_key: "42a0429606754917d221a0a69123dce9"
server_key_iv: "364a5a70a47c5f06e98f27415372df11"
server_key_version: "encryptionkey@0"
I20260923 17:34:37.301198 22914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:37.316447 22914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:37.321064 22914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:37.321338 22914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:37.324186 22914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:37.324285 22914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:37.324419 22914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:37.324493 22914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:34:37 dist-test-slave-ppjl krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184877, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:37.349713 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.324716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:47607 (local address 127.20.240.126:38733)
0923 17:34:37.324929 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:37.324935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:37.326411 (+ 1476us) server_negotiation.cc:407] Connection header received
0923 17:34:37.328714 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:37.328721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:37.328814 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:37.328934 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:37.331139 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.332167 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.335319 (+ 3152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.335849 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.336003 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.341764 (+ 5761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:37.341771 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:37.341777 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:37.341826 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:37.345470 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:37.346673 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:37.346680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:37.346683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:37.346835 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:37.347502 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:37.347509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:37.347512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:37.347937 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:37.348087 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:37.349110 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0923 17:34:37.349349 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 17:34:37.351187 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.325399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:38733 (local address 127.20.240.67:47607)
0923 17:34:37.326229 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:37.326270 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:37.328352 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:37.329205 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:37.329213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:37.329975 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:37.330923 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:37.330947 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.332396 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.332403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:37.335077 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:37.335100 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.336148 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.336157 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:37.336595 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.337499 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:37.337539 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:37.341343 (+ 3804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:37.345692 (+ 4349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:37.345704 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:37.345725 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:37.346491 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:37.346983 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:37.346988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:37.346992 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:37.347352 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:37.348085 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:37.348108 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:37.348605 (+ 497us) client_negotiation.cc:769] Sending connection context
0923 17:34:37.348942 (+ 337us) client_negotiation.cc:240] Negotiation successful
0923 17:34:37.349572 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":344,"threads_started":1}
I20260923 17:34:37.365381 22771 heartbeater.cc:503] Master 127.20.240.126:38733 was elected leader, sending a full tablet report...
I20260923 17:34:37.373558 22914 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:36865
I20260923 17:34:37.373656 23048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:36865 every 8 connection(s)
I20260923 17:34:37.374851 22914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:34:37.376809 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.362150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:38733 (local address 127.20.240.67:39387)
0923 17:34:37.363702 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:37.363726 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:37.363907 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:37.364258 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:37.364263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:37.365040 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:37.365352 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:37.365363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.366559 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.366565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:37.367800 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:37.367813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.368550 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.368557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:37.368723 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.369428 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:37.369448 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:37.370218 (+ 770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:37.374245 (+ 4027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:37.374251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:37.374255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:37.374683 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:37.375099 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:37.375104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:37.375108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:37.375279 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:37.375904 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:37.375909 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:37.376130 (+ 221us) client_negotiation.cc:769] Sending connection context
0923 17:34:37.376261 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 17:34:37.376517 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1374,"thread_start_us":1220,"threads_started":1}
I20260923 17:34:37.376992 23049 heartbeater.cc:347] Connected to a master server at 127.20.240.126:38733
I20260923 17:34:37.377211 23049 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:37.377693 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.362158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:39387 (local address 127.20.240.126:38733)
0923 17:34:37.362382 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:37.362388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:37.363846 (+ 1458us) server_negotiation.cc:407] Connection header received
0923 17:34:37.364032 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:37.364036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:37.364103 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:37.364197 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:37.365534 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.366402 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.367947 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.368358 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.368471 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.370409 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:37.370413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:37.370417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:37.370449 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:37.374101 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:37.374814 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:37.374820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:37.374824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:37.374985 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:37.375393 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:37.375399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:37.375403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:37.375735 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:37.377068 (+ 1333us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:37.377191 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 17:34:37.377435 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 17:34:37.377959 23049 heartbeater.cc:511] Master 127.20.240.126:38733 requested a full tablet report, sending...
I20260923 17:34:37.378640 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 22914
I20260923 17:34:37.378818 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:34:37.379432 21441 external_mini_cluster.cc:1428] Setting key 688a68bc2c5f633df80b8a8cbb09f6c3
I20260923 17:34:37.379503 22564 ts_manager.cc:194] Registered new tserver with Master: f19426ade30947f081be4568f1ab29e9 (127.20.240.67:36865)
I20260923 17:34:37.380505 22564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:47607
I20260923 17:34:37.383931 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:34:37 dist-test-slave-ppjl krb5kdc[22519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184877, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:34:37 dist-test-slave-ppjl krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184877, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:37.428539 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.414627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.20.240.126:38733)
0923 17:34:37.414794 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:37.414800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:37.415111 (+ 311us) server_negotiation.cc:407] Connection header received
0923 17:34:37.415371 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:37.415376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:37.415446 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:37.415539 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:37.417008 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.417750 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.419118 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.419621 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.419727 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.423439 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:37.423455 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:37.423459 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:37.423497 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:37.425998 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:37.426682 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:37.426686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:37.426689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:37.426814 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:37.427291 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:37.427295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:37.427297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:37.427589 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:37.427786 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:37.428061 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 17:34:37.428243 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:34:37.432894 22563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:34:37.439913 22563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:34:37.475010 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.451853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59254 (local address 127.20.240.67:36865)
0923 17:34:37.463152 (+ 11299us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:37.463185 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:37.464511 (+ 1326us) server_negotiation.cc:407] Connection header received
0923 17:34:37.464623 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:37.464631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:37.464927 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:37.465146 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:37.466505 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.468029 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.472332 (+ 4303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.473759 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.473884 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.474128 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:37.474531 (+ 403us) server_negotiation.cc:299] Negotiation successful
0923 17:34:37.474696 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11047,"thread_start_us":11347,"threads_started":1}
I20260923 17:34:37.475186 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.451744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:39589 (local address 127.0.0.1:43988)
0923 17:34:37.466843 (+ 15099us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:37.466869 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:37.467013 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:37.468011 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:37.468018 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:37.468046 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:37.468560 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:37.468573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.470645 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.470651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:37.472919 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:37.472933 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.474288 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.474295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:37.474467 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.474499 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:34:37.474589 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 17:34:37.474729 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14964,"mutex_wait_us":36,"thread_start_us":11851,"threads_started":1}
I20260923 17:34:37.476053 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.451571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:36865 (local address 127.0.0.1:59254)
0923 17:34:37.464031 (+ 12460us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:37.464090 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:37.464286 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:37.465896 (+ 1610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:37.465902 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:37.465935 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:37.466267 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:37.466276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.468215 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.468221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:37.472168 (+ 3947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:37.472183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.473966 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.473973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:37.474160 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.474217 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 17:34:37.474344 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 17:34:37.474545 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12145,"mutex_wait_us":4,"thread_start_us":15490,"threads_started":1}
I20260923 17:34:37.476094 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.452396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.20.240.65:39589)
0923 17:34:37.465798 (+ 13402us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:37.465806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:37.467018 (+ 1212us) server_negotiation.cc:407] Connection header received
0923 17:34:37.467272 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:37.467282 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:37.467563 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:37.467765 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:37.468756 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.470427 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.473092 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.474099 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.474230 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.474419 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:37.475627 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0923 17:34:37.475804 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13266,"thread_start_us":13127,"threads_started":1}
I20260923 17:34:37.479338 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.467401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:36083 (local address 127.0.0.1:40538)
0923 17:34:37.471903 (+ 4502us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:37.471927 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:37.472058 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:37.472618 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:37.472623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:37.472646 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:37.472999 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:37.473010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.474676 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.474681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:37.476723 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:37.476739 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.478526 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.478532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:37.478707 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.478741 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:34:37.478838 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 17:34:37.478976 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4365,"thread_start_us":4208,"threads_started":1}
I20260923 17:34:37.479365 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.464971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40538 (local address 127.20.240.66:36083)
0923 17:34:37.471550 (+ 6579us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:37.471559 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:37.472049 (+ 490us) server_negotiation.cc:407] Connection header received
0923 17:34:37.472227 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:37.472235 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:37.472439 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:37.472613 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:37.473183 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.474525 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.476880 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.478314 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.478464 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.478654 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:37.478911 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 17:34:37.479071 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6422,"thread_start_us":6297,"threads_started":1}
I20260923 17:34:37.479730 22979 tablet_service.cc:1467] Processing CreateTablet for tablet 32cb5edcfa834275b39e6f3cb6700e68 (DEFAULT_TABLE table=test-table [id=5d76c4a0a48c4151aac351569505f03d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:37.479738 22701 tablet_service.cc:1467] Processing CreateTablet for tablet 32cb5edcfa834275b39e6f3cb6700e68 (DEFAULT_TABLE table=test-table [id=5d76c4a0a48c4151aac351569505f03d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:37.481098 22701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32cb5edcfa834275b39e6f3cb6700e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:37.481138 22979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32cb5edcfa834275b39e6f3cb6700e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:37.483294 22840 tablet_service.cc:1467] Processing CreateTablet for tablet 32cb5edcfa834275b39e6f3cb6700e68 (DEFAULT_TABLE table=test-table [id=5d76c4a0a48c4151aac351569505f03d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:37.484704 22840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32cb5edcfa834275b39e6f3cb6700e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:37.493597 23068 tablet_bootstrap.cc:492] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Bootstrap starting.
I20260923 17:34:37.495903 23069 tablet_bootstrap.cc:492] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9: Bootstrap starting.
I20260923 17:34:37.497707 23068 tablet_bootstrap.cc:654] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:37.499037 23068 log.cc:824] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:37.499086 23070 tablet_bootstrap.cc:492] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e: Bootstrap starting.
I20260923 17:34:37.499470 23069 tablet_bootstrap.cc:654] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:37.500594 23069 log.cc:824] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:37.503047 23070 tablet_bootstrap.cc:654] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:37.503551 23068 tablet_bootstrap.cc:492] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb: No bootstrap required, opened a new log
I20260923 17:34:37.503671 23069 tablet_bootstrap.cc:492] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9: No bootstrap required, opened a new log
I20260923 17:34:37.503890 23069 ts_tablet_manager.cc:1397] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260923 17:34:37.503922 23068 ts_tablet_manager.cc:1397] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 17:34:37.504225 23070 log.cc:824] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:37.507020 23070 tablet_bootstrap.cc:492] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e: No bootstrap required, opened a new log
I20260923 17:34:37.507318 23070 ts_tablet_manager.cc:1397] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260923 17:34:37.514716 23070 raft_consensus.cc:348] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } }
I20260923 17:34:37.514789 23068 raft_consensus.cc:348] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } }
I20260923 17:34:37.514781 23069 raft_consensus.cc:348] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } }
I20260923 17:34:37.515128 23070 raft_consensus.cc:374] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:37.515249 23070 raft_consensus.cc:729] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2979bbaf9f264d79a6488d364451210e, State: Initialized, Role: FOLLOWER
I20260923 17:34:37.515264 23069 raft_consensus.cc:374] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:37.515264 23068 raft_consensus.cc:374] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:37.515362 23069 raft_consensus.cc:729] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19426ade30947f081be4568f1ab29e9, State: Initialized, Role: FOLLOWER
I20260923 17:34:37.515372 23068 raft_consensus.cc:729] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9908ba2c3f2f4b88a520c0ea8fa8edcb, State: Initialized, Role: FOLLOWER
I20260923 17:34:37.516003 23069 consensus_queue.cc:260] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } }
I20260923 17:34:37.516003 23068 consensus_queue.cc:260] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } }
I20260923 17:34:37.516002 23070 consensus_queue.cc:260] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } }
I20260923 17:34:37.518489 23049 heartbeater.cc:503] Master 127.20.240.126:38733 was elected leader, sending a full tablet report...
I20260923 17:34:37.519788 23068 ts_tablet_manager.cc:1428] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Time spent starting tablet: real 0.016s user 0.005s sys 0.010s
I20260923 17:34:37.519928 23070 ts_tablet_manager.cc:1428] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260923 17:34:37.520010 22910 heartbeater.cc:503] Master 127.20.240.126:38733 was elected leader, sending a full tablet report...
I20260923 17:34:37.521523 23069 ts_tablet_manager.cc:1428] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9: Time spent starting tablet: real 0.018s user 0.014s sys 0.001s
W20260923 17:34:37.607399 22772 tablet.cc:2369] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:34:37.623734 22911 tablet.cc:2369] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:34:37.626612 23050 tablet.cc:2369] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:34:37.793877 23074 raft_consensus.cc:482] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:34:37.794189 23074 raft_consensus.cc:504] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } }
I20260923 17:34:37.795969 23074 leader_election.cc:288] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9908ba2c3f2f4b88a520c0ea8fa8edcb (127.20.240.65:39589), 2979bbaf9f264d79a6488d364451210e (127.20.240.66:36083)
I20260923 17:34:37.801906 22938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.796406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:39589 (local address 127.20.240.67:50085)
0923 17:34:37.796616 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:37.796638 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:37.796785 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:37.797237 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:37.797242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:37.797285 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:37.797521 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:37.797537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.798812 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.798817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:37.800105 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:37.800117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.801191 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.801196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:37.801368 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.801393 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:34:37.801479 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 17:34:37.801614 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 17:34:37.801898 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.796576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:50085 (local address 127.20.240.65:39589)
0923 17:34:37.796792 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:37.796798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:37.796819 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:34:37.796892 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:37.796897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:37.796981 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:37.797067 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:37.797683 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.798635 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.800274 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.801042 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.801199 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.801273 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:37.801515 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 17:34:37.801640 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:34:37.802561 23077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.796817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:36083 (local address 127.20.240.67:54073)
0923 17:34:37.797678 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:37.797692 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:37.797779 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:37.798054 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:37.798057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:37.798074 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:37.798368 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:37.798376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.799434 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.799439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:37.800736 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:37.800747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.801898 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.801905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:37.802053 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.802089 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:34:37.802174 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 17:34:37.802297 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":543,"threads_started":1}
I20260923 17:34:37.802597 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.797079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:54073 (local address 127.20.240.66:36083)
0923 17:34:37.797283 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:37.797303 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:37.797782 (+ 479us) server_negotiation.cc:407] Connection header received
0923 17:34:37.797862 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:37.797867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:37.797945 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:37.798045 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:37.798523 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.799293 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.800894 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.801669 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.801813 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.801885 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:37.802214 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 17:34:37.802351 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:34:37.803105 22721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cb5edcfa834275b39e6f3cb6700e68" candidate_uuid: "f19426ade30947f081be4568f1ab29e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" is_pre_election: true
I20260923 17:34:37.803644 22721 raft_consensus.cc:2454] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19426ade30947f081be4568f1ab29e9 in term 0.
I20260923 17:34:37.803622 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cb5edcfa834275b39e6f3cb6700e68" candidate_uuid: "f19426ade30947f081be4568f1ab29e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2979bbaf9f264d79a6488d364451210e" is_pre_election: true
I20260923 17:34:37.804087 22860 raft_consensus.cc:2454] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19426ade30947f081be4568f1ab29e9 in term 0.
I20260923 17:34:37.804388 22931 leader_election.cc:302] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9908ba2c3f2f4b88a520c0ea8fa8edcb, f19426ade30947f081be4568f1ab29e9; no voters:
I20260923 17:34:37.804814 23074 raft_consensus.cc:2781] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:34:37.804963 23074 raft_consensus.cc:482] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:34:37.805076 23074 raft_consensus.cc:3037] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:37.807423 23074 raft_consensus.cc:504] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } }
I20260923 17:34:37.808003 23074 leader_election.cc:288] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [CANDIDATE]: Term 1 election: Requested vote from peers 9908ba2c3f2f4b88a520c0ea8fa8edcb (127.20.240.65:39589), 2979bbaf9f264d79a6488d364451210e (127.20.240.66:36083)
I20260923 17:34:37.808549 22721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cb5edcfa834275b39e6f3cb6700e68" candidate_uuid: "f19426ade30947f081be4568f1ab29e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb"
I20260923 17:34:37.808593 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cb5edcfa834275b39e6f3cb6700e68" candidate_uuid: "f19426ade30947f081be4568f1ab29e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2979bbaf9f264d79a6488d364451210e"
I20260923 17:34:37.808745 22721 raft_consensus.cc:3037] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:37.808774 22860 raft_consensus.cc:3037] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:37.811015 22721 raft_consensus.cc:2454] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19426ade30947f081be4568f1ab29e9 in term 1.
I20260923 17:34:37.811020 22860 raft_consensus.cc:2454] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19426ade30947f081be4568f1ab29e9 in term 1.
I20260923 17:34:37.811465 22931 leader_election.cc:302] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2979bbaf9f264d79a6488d364451210e, f19426ade30947f081be4568f1ab29e9; no voters:
I20260923 17:34:37.811797 23074 raft_consensus.cc:2781] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:37.812232 23074 raft_consensus.cc:686] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [term 1 LEADER]: Becoming Leader. State: Replica: f19426ade30947f081be4568f1ab29e9, State: Running, Role: LEADER
I20260923 17:34:37.812712 23074 consensus_queue.cc:237] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } }
I20260923 17:34:37.817612 22563 catalog_manager.cc:5495] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f19426ade30947f081be4568f1ab29e9 (127.20.240.67). New cstate: current_term: 1 leader_uuid: "f19426ade30947f081be4568f1ab29e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } health_report { overall_health: UNKNOWN } } }
I20260923 17:34:37.928579 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:37.922778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59262 (local address 127.20.240.67:36865)
0923 17:34:37.923000 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:37.923006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:37.923095 (+ 89us) server_negotiation.cc:407] Connection header received
0923 17:34:37.923515 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:37.923519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:37.923597 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:37.923717 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:34:37.924098 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.924790 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.926205 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:37.926615 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:37.926733 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:37.927333 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:34:37.927896 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:34:37.928074 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:37.928218 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 17:34:37.928311 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:34:37.936830 22721 raft_consensus.cc:1261] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 1 FOLLOWER]: Refusing update from remote peer f19426ade30947f081be4568f1ab29e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:34:37.936831 22860 raft_consensus.cc:1261] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e [term 1 FOLLOWER]: Refusing update from remote peer f19426ade30947f081be4568f1ab29e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:34:37.937765 23078 consensus_queue.cc:1035] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:34:37.938135 23074 consensus_queue.cc:1035] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:34:37.948812 23081 mvcc.cc:205] Tried to move back new op lower bound from 7332597260018733056 to 7332597259524804608. Current Snapshot: MvccSnapshot[applied={T|T < 7332597260018733056}]
I20260923 17:34:37.955765 23083 mvcc.cc:205] Tried to move back new op lower bound from 7332597260018733056 to 7332597259524804608. Current Snapshot: MvccSnapshot[applied={T|T < 7332597260018733056}]
I20260923 17:34:37.971969 22563 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:34:37.972221 22563 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:34:37.976181 22563 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d: Sending DeleteTablet for 3 replicas of tablet 32cb5edcfa834275b39e6f3cb6700e68
I20260923 17:34:37.977540 22840 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cb5edcfa834275b39e6f3cb6700e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:37 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:40538
I20260923 17:34:37.977525 22979 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cb5edcfa834275b39e6f3cb6700e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:37 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:59254
I20260923 17:34:37.977551 22701 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cb5edcfa834275b39e6f3cb6700e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:37 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:43988
I20260923 17:34:37.979385 22563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:34:37.980181 22563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:34:37.989339 22839 tablet_service.cc:1467] Processing CreateTablet for tablet 5a97f61010504d15a334c2564939f5ef (DEFAULT_TABLE table=test-table [id=019d0ecfa8fb42efbd76cf50a1ee7d90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:37.990024 22839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a97f61010504d15a334c2564939f5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:37.991650 23092 tablet_replica.cc:330] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9: stopping tablet replica
I20260923 17:34:37.992146 23092 raft_consensus.cc:2229] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:34:37.992812 23092 raft_consensus.cc:2258] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:37.993614 23091 tablet_replica.cc:330] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb: stopping tablet replica
I20260923 17:34:37.994088 23091 raft_consensus.cc:2229] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:34:37.994587 23091 raft_consensus.cc:2258] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:37.996018 22700 tablet_service.cc:1467] Processing CreateTablet for tablet 5a97f61010504d15a334c2564939f5ef (DEFAULT_TABLE table=test-table [id=019d0ecfa8fb42efbd76cf50a1ee7d90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:37.996479 22700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a97f61010504d15a334c2564939f5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:37.996968 23092 ts_tablet_manager.cc:1905] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:37.996968 23091 ts_tablet_manager.cc:1905] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:37.999537 22979 tablet_service.cc:1467] Processing CreateTablet for tablet 5a97f61010504d15a334c2564939f5ef (DEFAULT_TABLE table=test-table [id=019d0ecfa8fb42efbd76cf50a1ee7d90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:37.999989 22979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a97f61010504d15a334c2564939f5ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:38.005473 23068 tablet_bootstrap.cc:492] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Bootstrap starting.
I20260923 17:34:38.006835 23070 tablet_bootstrap.cc:492] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e: Bootstrap starting.
I20260923 17:34:38.007143 23068 tablet_bootstrap.cc:654] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:38.008472 23070 tablet_bootstrap.cc:654] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:38.008455 23091 ts_tablet_manager.cc:1918] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:34:38.008778 23091 log.cc:1196] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/32cb5edcfa834275b39e6f3cb6700e68
I20260923 17:34:38.009416 23091 ts_tablet_manager.cc:1939] T 32cb5edcfa834275b39e6f3cb6700e68 P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Deleting consensus metadata
I20260923 17:34:38.011770 22549 catalog_manager.cc:4841] TS 9908ba2c3f2f4b88a520c0ea8fa8edcb (127.20.240.65:39589): tablet 32cb5edcfa834275b39e6f3cb6700e68 (table test-table [id=5d76c4a0a48c4151aac351569505f03d]) successfully deleted
I20260923 17:34:38.015393 23070 tablet_bootstrap.cc:492] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e: No bootstrap required, opened a new log
I20260923 17:34:38.015682 23070 ts_tablet_manager.cc:1397] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260923 17:34:38.016640 23070 raft_consensus.cc:348] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.017015 23070 raft_consensus.cc:374] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:38.017151 23070 raft_consensus.cc:729] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2979bbaf9f264d79a6488d364451210e, State: Initialized, Role: FOLLOWER
I20260923 17:34:38.017517 23070 consensus_queue.cc:260] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.018278 23070 ts_tablet_manager.cc:1428] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:34:38.018301 23092 ts_tablet_manager.cc:1918] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:34:38.018517 23092 log.cc:1196] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/32cb5edcfa834275b39e6f3cb6700e68
I20260923 17:34:38.019618 23090 tablet_replica.cc:330] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e: stopping tablet replica
I20260923 17:34:38.020092 23090 raft_consensus.cc:2229] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:34:38.020601 23090 raft_consensus.cc:2258] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:38.023842 23068 tablet_bootstrap.cc:492] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb: No bootstrap required, opened a new log
I20260923 17:34:38.023998 23068 ts_tablet_manager.cc:1397] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.005s
I20260923 17:34:38.024076 23090 ts_tablet_manager.cc:1905] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:38.024713 23068 raft_consensus.cc:348] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.024931 23068 raft_consensus.cc:374] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:38.025018 23068 raft_consensus.cc:729] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9908ba2c3f2f4b88a520c0ea8fa8edcb, State: Initialized, Role: FOLLOWER
I20260923 17:34:38.025314 23068 consensus_queue.cc:260] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.025965 23068 ts_tablet_manager.cc:1428] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:34:38.026607 23095 tablet_bootstrap.cc:492] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9: Bootstrap starting.
I20260923 17:34:38.028573 23095 tablet_bootstrap.cc:654] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:38.031195 23092 ts_tablet_manager.cc:1939] T 32cb5edcfa834275b39e6f3cb6700e68 P f19426ade30947f081be4568f1ab29e9: Deleting consensus metadata
I20260923 17:34:38.033231 22550 catalog_manager.cc:4841] TS f19426ade30947f081be4568f1ab29e9 (127.20.240.67:36865): tablet 32cb5edcfa834275b39e6f3cb6700e68 (table test-table [id=5d76c4a0a48c4151aac351569505f03d]) successfully deleted
I20260923 17:34:38.033428 23090 ts_tablet_manager.cc:1918] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:34:38.033603 23090 log.cc:1196] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/32cb5edcfa834275b39e6f3cb6700e68
I20260923 17:34:38.034127 23090 ts_tablet_manager.cc:1939] T 32cb5edcfa834275b39e6f3cb6700e68 P 2979bbaf9f264d79a6488d364451210e: Deleting consensus metadata
I20260923 17:34:38.036055 22549 catalog_manager.cc:4841] TS 2979bbaf9f264d79a6488d364451210e (127.20.240.66:36083): tablet 32cb5edcfa834275b39e6f3cb6700e68 (table test-table [id=5d76c4a0a48c4151aac351569505f03d]) successfully deleted
I20260923 17:34:38.037086 23095 tablet_bootstrap.cc:492] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9: No bootstrap required, opened a new log
I20260923 17:34:38.037240 23095 ts_tablet_manager.cc:1397] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260923 17:34:38.037962 23095 raft_consensus.cc:348] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.038199 23095 raft_consensus.cc:374] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:38.038304 23095 raft_consensus.cc:729] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19426ade30947f081be4568f1ab29e9, State: Initialized, Role: FOLLOWER
I20260923 17:34:38.038543 23095 consensus_queue.cc:260] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.039063 23095 ts_tablet_manager.cc:1428] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260923 17:34:38.069523 23076 raft_consensus.cc:482] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:34:38.069790 23076 raft_consensus.cc:504] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.071935 23076 leader_election.cc:288] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9908ba2c3f2f4b88a520c0ea8fa8edcb (127.20.240.65:39589), f19426ade30947f081be4568f1ab29e9 (127.20.240.67:36865)
I20260923 17:34:38.080363 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.072677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:44337 (local address 127.20.240.65:39589)
0923 17:34:38.072877 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:38.072884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:38.073503 (+ 619us) server_negotiation.cc:407] Connection header received
0923 17:34:38.073828 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:38.073834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:38.073922 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:38.074021 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:38.074752 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.075662 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.077940 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.079242 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.079418 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.079502 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:38.079960 (+ 458us) server_negotiation.cc:299] Negotiation successful
0923 17:34:38.080090 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:34:38.080528 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.072927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:59743 (local address 127.20.240.67:36865)
0923 17:34:38.073112 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:38.073119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:38.074025 (+ 906us) server_negotiation.cc:407] Connection header received
0923 17:34:38.074222 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:38.074227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:38.074287 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:38.074333 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:38.075103 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.076318 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.078537 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.079422 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.079590 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.079662 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:38.080077 (+ 415us) server_negotiation.cc:299] Negotiation successful
0923 17:34:38.080224 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:34:38.080581 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.072176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:39589 (local address 127.20.240.66:44337)
0923 17:34:38.073394 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:38.073418 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:38.073619 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:38.074125 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:38.074130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:38.074160 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:38.074515 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.074528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.075803 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.075808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.077756 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.077772 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.079481 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.079488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.079680 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.079770 (+ 90us) client_negotiation.cc:769] Sending connection context
0923 17:34:38.079890 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 17:34:38.080037 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"mutex_wait_us":19,"thread_start_us":920,"threads_started":1}
I20260923 17:34:38.081326 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.072732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:36865 (local address 127.20.240.66:59743)
0923 17:34:38.073898 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:38.073915 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:38.074053 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:38.074499 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:38.074502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:38.074521 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:38.074901 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.074912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.076492 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.076497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.078382 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.078394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.079679 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.079686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.079880 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.079914 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:34:38.080776 (+ 862us) client_negotiation.cc:240] Negotiation successful
0923 17:34:38.080927 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"mutex_wait_us":1,"thread_start_us":844,"threads_started":1}
I20260923 17:34:38.081437 22721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a97f61010504d15a334c2564939f5ef" candidate_uuid: "2979bbaf9f264d79a6488d364451210e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" is_pre_election: true
I20260923 17:34:38.081748 22721 raft_consensus.cc:2454] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2979bbaf9f264d79a6488d364451210e in term 0.
I20260923 17:34:38.082332 22792 leader_election.cc:302] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2979bbaf9f264d79a6488d364451210e, 9908ba2c3f2f4b88a520c0ea8fa8edcb; no voters:
I20260923 17:34:38.082355 22999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a97f61010504d15a334c2564939f5ef" candidate_uuid: "2979bbaf9f264d79a6488d364451210e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19426ade30947f081be4568f1ab29e9" is_pre_election: true
I20260923 17:34:38.082799 23076 raft_consensus.cc:2781] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:34:38.082788 22999 raft_consensus.cc:2454] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2979bbaf9f264d79a6488d364451210e in term 0.
I20260923 17:34:38.082926 23076 raft_consensus.cc:482] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:34:38.083024 23076 raft_consensus.cc:3037] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:38.086275 23076 raft_consensus.cc:504] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.087097 23076 leader_election.cc:288] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [CANDIDATE]: Term 1 election: Requested vote from peers 9908ba2c3f2f4b88a520c0ea8fa8edcb (127.20.240.65:39589), f19426ade30947f081be4568f1ab29e9 (127.20.240.67:36865)
I20260923 17:34:38.087728 22999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a97f61010504d15a334c2564939f5ef" candidate_uuid: "2979bbaf9f264d79a6488d364451210e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19426ade30947f081be4568f1ab29e9"
I20260923 17:34:38.087785 22721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a97f61010504d15a334c2564939f5ef" candidate_uuid: "2979bbaf9f264d79a6488d364451210e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb"
I20260923 17:34:38.087934 22999 raft_consensus.cc:3037] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:38.088025 22721 raft_consensus.cc:3037] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:38.090384 22999 raft_consensus.cc:2454] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2979bbaf9f264d79a6488d364451210e in term 1.
I20260923 17:34:38.090906 22793 leader_election.cc:302] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2979bbaf9f264d79a6488d364451210e, f19426ade30947f081be4568f1ab29e9; no voters:
I20260923 17:34:38.091025 22721 raft_consensus.cc:2454] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2979bbaf9f264d79a6488d364451210e in term 1.
I20260923 17:34:38.091260 23076 raft_consensus.cc:2781] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:38.092125 23076 raft_consensus.cc:686] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [term 1 LEADER]: Becoming Leader. State: Replica: 2979bbaf9f264d79a6488d364451210e, State: Running, Role: LEADER
I20260923 17:34:38.092662 23076 consensus_queue.cc:237] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.096457 22563 catalog_manager.cc:5495] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2979bbaf9f264d79a6488d364451210e (127.20.240.66). New cstate: current_term: 1 leader_uuid: "2979bbaf9f264d79a6488d364451210e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } health_report { overall_health: UNKNOWN } } }
Sep 23 17:34:38 dist-test-slave-ppjl krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184875, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:38.162885 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.145692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.20.240.126:38733)
0923 17:34:38.147622 (+ 1930us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:38.147627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:38.147647 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:34:38.147699 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:38.147703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:38.147792 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:38.147888 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:38.149418 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.150236 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.152084 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.152488 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.152585 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.156689 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:38.156697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:38.156702 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:38.156742 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:38.159845 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:38.160808 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:38.160812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:38.160814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:38.160946 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:38.161366 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:38.161369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:38.161371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:38.161647 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:38.161752 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:38.162290 (+ 538us) server_negotiation.cc:299] Negotiation successful
0923 17:34:38.162549 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1794,"thread_start_us":1663,"threads_started":1}
I20260923 17:34:38.163314 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.145628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:38733 (local address 127.0.0.1:37552)
0923 17:34:38.146322 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:38.146339 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:38.146478 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:38.147997 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:38.148003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:38.148908 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:38.149207 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.149219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.150387 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.150392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.151928 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.151946 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.152674 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.152681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.152862 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.153641 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:38.153665 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:38.156438 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:38.160000 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:38.160019 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:38.160028 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:38.160669 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:38.161076 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:38.161082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:38.161086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:38.161262 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:38.161782 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:38.161792 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:38.162000 (+ 208us) client_negotiation.cc:769] Sending connection context
0923 17:34:38.162205 (+ 205us) client_negotiation.cc:240] Negotiation successful
0923 17:34:38.162720 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":183,"thread_start_us":361,"threads_started":1}
I20260923 17:34:38.169162 22563 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:37552:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 17:34:38.170444 22563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:34:38.178113 22701 tablet_service.cc:1467] Processing CreateTablet for tablet ca9870ce500448d7a2a7ecbfc086b0df (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=333a2b9f6df74e9cb0a7a9215f9ed042]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:34:38.178248 22840 tablet_service.cc:1467] Processing CreateTablet for tablet ca9870ce500448d7a2a7ecbfc086b0df (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=333a2b9f6df74e9cb0a7a9215f9ed042]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:34:38.178387 22979 tablet_service.cc:1467] Processing CreateTablet for tablet ca9870ce500448d7a2a7ecbfc086b0df (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=333a2b9f6df74e9cb0a7a9215f9ed042]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:34:38.178802 22701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca9870ce500448d7a2a7ecbfc086b0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:38.178815 22840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca9870ce500448d7a2a7ecbfc086b0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:38.178886 22979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca9870ce500448d7a2a7ecbfc086b0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:38.186458 23068 tablet_bootstrap.cc:492] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Bootstrap starting.
I20260923 17:34:38.186569 23070 tablet_bootstrap.cc:492] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e: Bootstrap starting.
I20260923 17:34:38.188037 23070 tablet_bootstrap.cc:654] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:38.188248 23068 tablet_bootstrap.cc:654] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:38.190142 23095 tablet_bootstrap.cc:492] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9: Bootstrap starting.
I20260923 17:34:38.191597 23070 tablet_bootstrap.cc:492] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e: No bootstrap required, opened a new log
I20260923 17:34:38.191774 23070 ts_tablet_manager.cc:1397] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260923 17:34:38.192056 23095 tablet_bootstrap.cc:654] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:38.192668 23070 raft_consensus.cc:348] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.192912 23070 raft_consensus.cc:374] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:38.192991 23070 raft_consensus.cc:729] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2979bbaf9f264d79a6488d364451210e, State: Initialized, Role: FOLLOWER
I20260923 17:34:38.193418 23070 consensus_queue.cc:260] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.193961 23076 tablet_replica.cc:439] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } }
I20260923 17:34:38.194265 23076 tablet_replica.cc:442] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:34:38.197506 23070 ts_tablet_manager.cc:1428] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260923 17:34:38.200575 23068 tablet_bootstrap.cc:492] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb: No bootstrap required, opened a new log
I20260923 17:34:38.200774 23068 ts_tablet_manager.cc:1397] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260923 17:34:38.201587 23068 raft_consensus.cc:348] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.201879 23068 raft_consensus.cc:374] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:38.201956 23068 raft_consensus.cc:729] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9908ba2c3f2f4b88a520c0ea8fa8edcb, State: Initialized, Role: FOLLOWER
I20260923 17:34:38.202117 23095 tablet_bootstrap.cc:492] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9: No bootstrap required, opened a new log
I20260923 17:34:38.202314 23095 ts_tablet_manager.cc:1397] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260923 17:34:38.202253 23068 consensus_queue.cc:260] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.202733 23075 tablet_replica.cc:439] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } }
I20260923 17:34:38.203246 23075 tablet_replica.cc:442] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:34:38.203204 23095 raft_consensus.cc:348] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.203725 23068 ts_tablet_manager.cc:1428] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260923 17:34:38.203513 23095 raft_consensus.cc:374] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:38.204694 23095 raft_consensus.cc:729] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19426ade30947f081be4568f1ab29e9, State: Initialized, Role: FOLLOWER
I20260923 17:34:38.205036 23095 consensus_queue.cc:260] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.205542 23074 tablet_replica.cc:439] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } }
I20260923 17:34:38.205833 23074 tablet_replica.cc:442] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:34:38.206931 23095 ts_tablet_manager.cc:1428] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260923 17:34:38.380074 23075 raft_consensus.cc:482] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:34:38.380297 23075 raft_consensus.cc:504] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.381949 23075 leader_election.cc:288] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2979bbaf9f264d79a6488d364451210e (127.20.240.66:36083), f19426ade30947f081be4568f1ab29e9 (127.20.240.67:36865)
I20260923 17:34:38.402419 23111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.382235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:36083 (local address 127.20.240.65:34137)
0923 17:34:38.391108 (+ 8873us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:38.391133 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:38.391513 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:38.393092 (+ 1579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:38.393097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:38.393121 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:38.393670 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.393685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.396520 (+ 2835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.396526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.399238 (+ 2712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.399264 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.401313 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.401321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.401642 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.401728 (+ 86us) client_negotiation.cc:769] Sending connection context
0923 17:34:38.401827 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 17:34:38.401998 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8706,"mutex_wait_us":121,"thread_start_us":8933,"threads_started":1}
I20260923 17:34:38.402365 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.382973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:55181 (local address 127.20.240.67:36865)
0923 17:34:38.383221 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:38.383228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:38.392933 (+ 9705us) server_negotiation.cc:407] Connection header received
0923 17:34:38.393044 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:38.393049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:38.393129 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:38.393215 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:38.394642 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.396304 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.399489 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.400917 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.401079 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.401151 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:38.401908 (+ 757us) server_negotiation.cc:299] Negotiation successful
0923 17:34:38.402065 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:34:38.403215 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.382856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:34137 (local address 127.20.240.66:36083)
0923 17:34:38.391996 (+ 9140us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:38.392003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:38.392025 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:34:38.392105 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:38.392111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:38.392195 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:38.393692 (+ 1497us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:38.394761 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.396302 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.399494 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.400764 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.400995 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.401092 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:38.402691 (+ 1599us) server_negotiation.cc:299] Negotiation successful
0923 17:34:38.402866 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9007,"thread_start_us":8879,"threads_started":1}
I20260923 17:34:38.403826 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.382434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:36865 (local address 127.20.240.65:55181)
0923 17:34:38.392788 (+ 10354us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:38.392804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:38.392932 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:38.393518 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:38.393523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:38.393554 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:38.394299 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.394311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.396520 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.396526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.399238 (+ 2712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.399263 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.401350 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.401355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.401647 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.401729 (+ 82us) client_negotiation.cc:769] Sending connection context
0923 17:34:38.403094 (+ 1365us) client_negotiation.cc:240] Negotiation successful
0923 17:34:38.403475 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10248,"mutex_wait_us":451,"thread_start_us":10035,"threads_started":1}
I20260923 17:34:38.403887 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9870ce500448d7a2a7ecbfc086b0df" candidate_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2979bbaf9f264d79a6488d364451210e" is_pre_election: true
I20260923 17:34:38.404179 22860 raft_consensus.cc:2454] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9908ba2c3f2f4b88a520c0ea8fa8edcb in term 0.
I20260923 17:34:38.404999 22999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9870ce500448d7a2a7ecbfc086b0df" candidate_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19426ade30947f081be4568f1ab29e9" is_pre_election: true
I20260923 17:34:38.405191 22652 leader_election.cc:302] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2979bbaf9f264d79a6488d364451210e, 9908ba2c3f2f4b88a520c0ea8fa8edcb; no voters:
I20260923 17:34:38.405274 22999 raft_consensus.cc:2454] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9908ba2c3f2f4b88a520c0ea8fa8edcb in term 0.
I20260923 17:34:38.405692 23075 raft_consensus.cc:2781] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:34:38.405794 23075 raft_consensus.cc:482] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:34:38.405905 23075 raft_consensus.cc:3037] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:38.408136 23075 raft_consensus.cc:504] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.408723 23075 leader_election.cc:288] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [CANDIDATE]: Term 1 election: Requested vote from peers 2979bbaf9f264d79a6488d364451210e (127.20.240.66:36083), f19426ade30947f081be4568f1ab29e9 (127.20.240.67:36865)
I20260923 17:34:38.409363 22860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9870ce500448d7a2a7ecbfc086b0df" candidate_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2979bbaf9f264d79a6488d364451210e"
I20260923 17:34:38.409607 22860 raft_consensus.cc:3037] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:38.409605 22999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca9870ce500448d7a2a7ecbfc086b0df" candidate_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19426ade30947f081be4568f1ab29e9"
I20260923 17:34:38.409838 22999 raft_consensus.cc:3037] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:38.412655 22860 raft_consensus.cc:2454] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9908ba2c3f2f4b88a520c0ea8fa8edcb in term 1.
I20260923 17:34:38.412945 22999 raft_consensus.cc:2454] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9908ba2c3f2f4b88a520c0ea8fa8edcb in term 1.
I20260923 17:34:38.413280 22652 leader_election.cc:302] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2979bbaf9f264d79a6488d364451210e, 9908ba2c3f2f4b88a520c0ea8fa8edcb; no voters:
I20260923 17:34:38.413679 23075 raft_consensus.cc:2781] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:38.414206 23075 raft_consensus.cc:686] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 1 LEADER]: Becoming Leader. State: Replica: 9908ba2c3f2f4b88a520c0ea8fa8edcb, State: Running, Role: LEADER
I20260923 17:34:38.414822 23075 consensus_queue.cc:237] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } }
I20260923 17:34:38.416893 23114 tablet_replica.cc:439] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb: TxnStatusTablet state changed. Reason: New leader 9908ba2c3f2f4b88a520c0ea8fa8edcb. Latest consensus state: current_term: 1 leader_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } }
I20260923 17:34:38.417497 23114 tablet_replica.cc:442] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:34:38.418303 23116 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 17:34:38.418519 23116 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 17:34:38.418879 22564 catalog_manager.cc:5495] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9908ba2c3f2f4b88a520c0ea8fa8edcb (127.20.240.65). New cstate: current_term: 1 leader_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } health_report { overall_health: UNKNOWN } } }
I20260923 17:34:38.449963 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.444560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.20.240.65:39589)
0923 17:34:38.444737 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:38.444743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:38.444764 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:34:38.444851 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:38.444855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:38.444932 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:38.445050 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:34:38.445430 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.446202 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.447613 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.448050 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.448165 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.448632 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:34:38.449205 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:34:38.449339 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:38.449537 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 17:34:38.449623 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:34:38.449951 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.444194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:39589 (local address 127.0.0.1:43990)
0923 17:34:38.444423 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:38.444444 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:38.444586 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:38.445105 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:38.445109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:38.445122 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:34:38.445285 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.445294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.446360 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.446365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.447457 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.447469 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.448214 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.448220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.448393 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.448452 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:34:38.449396 (+ 944us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:34:38.449413 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 17:34:38.449485 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 17:34:38.449651 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 17:34:38.454527 22860 raft_consensus.cc:1261] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e [term 1 FOLLOWER]: Refusing update from remote peer 9908ba2c3f2f4b88a520c0ea8fa8edcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:34:38.454821 22999 raft_consensus.cc:1261] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9 [term 1 FOLLOWER]: Refusing update from remote peer 9908ba2c3f2f4b88a520c0ea8fa8edcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:34:38.455752 23075 consensus_queue.cc:1035] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:34:38.456661 23075 consensus_queue.cc:1035] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260923 17:34:38.472438 23076 tablet_replica.cc:439] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e: TxnStatusTablet state changed. Reason: New leader 9908ba2c3f2f4b88a520c0ea8fa8edcb. Latest consensus state: current_term: 1 leader_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } }
I20260923 17:34:38.473021 23076 tablet_replica.cc:442] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:34:38.477455 23074 tablet_replica.cc:439] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9: TxnStatusTablet state changed. Reason: New leader 9908ba2c3f2f4b88a520c0ea8fa8edcb. Latest consensus state: current_term: 1 leader_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } }
I20260923 17:34:38.477779 23074 tablet_replica.cc:442] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:34:38.481484 23074 tablet_replica.cc:439] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } }
I20260923 17:34:38.481751 23074 tablet_replica.cc:442] T ca9870ce500448d7a2a7ecbfc086b0df P f19426ade30947f081be4568f1ab29e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:34:38.482692 23120 mvcc.cc:205] Tried to move back new op lower bound from 7332597262142210048 to 7332597261991174144. Current Snapshot: MvccSnapshot[applied={T|T < 7332597262142210048}]
I20260923 17:34:38.488592 23076 tablet_replica.cc:439] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } }
I20260923 17:34:38.488932 23076 tablet_replica.cc:442] T ca9870ce500448d7a2a7ecbfc086b0df P 2979bbaf9f264d79a6488d364451210e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:34:38.489946 23075 tablet_replica.cc:439] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } }
I20260923 17:34:38.489887 23123 tablet_replica.cc:439] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 } } peers { permanent_uuid: "2979bbaf9f264d79a6488d364451210e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36083 } } peers { permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 } } }
I20260923 17:34:38.490290 23123 tablet_replica.cc:442] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:34:38.490288 23075 tablet_replica.cc:442] T ca9870ce500448d7a2a7ecbfc086b0df P 9908ba2c3f2f4b88a520c0ea8fa8edcb: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:34:38.504267 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.498199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.20.240.66:36083)
0923 17:34:38.498443 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:38.498450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:38.498975 (+ 525us) server_negotiation.cc:407] Connection header received
0923 17:34:38.499212 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:38.499218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:38.499314 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:38.499451 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:34:38.499928 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.500640 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.502045 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.502439 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.502544 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.502954 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:34:38.503548 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:34:38.503699 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:38.503840 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 17:34:38.503924 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:34:38.522540 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.515655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:34085 (local address 127.20.240.65:39589)
0923 17:34:38.515954 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:38.515961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:38.516494 (+ 533us) server_negotiation.cc:407] Connection header received
0923 17:34:38.516726 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:38.516731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:38.516806 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:38.516905 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:34:38.517617 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.518410 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.520535 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.521106 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.521221 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.521812 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:34:38.521960 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:34:38.522026 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:38.522192 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 17:34:38.522268 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:34:38.522722 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.515621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:39589 (local address 127.20.240.66:34085)
0923 17:34:38.516334 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:38.516357 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:38.516543 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:38.516970 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:38.516975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:38.516990 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:34:38.517360 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.517373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.518580 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.518586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.520359 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.520379 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.521316 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.521324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.521520 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.521608 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:34:38.522061 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:34:38.522073 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 17:34:38.522157 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:34:38.522328 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":349,"threads_started":1}
I20260923 17:34:38.532776 22999 raft_consensus.cc:1261] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9 [term 1 FOLLOWER]: Refusing update from remote peer 2979bbaf9f264d79a6488d364451210e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:34:38.532783 22721 raft_consensus.cc:1261] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 1 FOLLOWER]: Refusing update from remote peer 2979bbaf9f264d79a6488d364451210e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:34:38.533483 23099 consensus_queue.cc:1035] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9908ba2c3f2f4b88a520c0ea8fa8edcb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 39589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:34:38.533828 23076 consensus_queue.cc:1035] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19426ade30947f081be4568f1ab29e9" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:34:38.581789 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.567467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:40343 (local address 127.20.240.66:36083)
0923 17:34:38.567636 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:38.567642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:38.575883 (+ 8241us) server_negotiation.cc:407] Connection header received
0923 17:34:38.576077 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:38.576083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:38.576189 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:38.576284 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:34:38.577111 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.577969 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.579598 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.580134 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.580266 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.580881 (+ 615us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:34:38.581085 (+ 204us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:34:38.581198 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:38.581394 (+ 196us) server_negotiation.cc:299] Negotiation successful
0923 17:34:38.581486 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:34:38.582079 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.567021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:36083 (local address 127.20.240.65:40343)
0923 17:34:38.575745 (+ 8724us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:38.575767 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:38.575927 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:38.576455 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:38.576459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:38.576472 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:34:38.576915 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.576931 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.578132 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.578138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.579444 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:38.579459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.580360 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.580368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:38.580564 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.580648 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:34:38.581249 (+ 601us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:34:38.581260 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 17:34:38.581356 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 17:34:38.581513 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8476,"thread_start_us":8327,"threads_started":1}
I20260923 17:34:38.623240 22564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:34:38.623584 22564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:34:38.627382 22564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb5e648d26874de98cae4a66cf55d66d: Sending DeleteTablet for 3 replicas of tablet 5a97f61010504d15a334c2564939f5ef
I20260923 17:34:38.628494 22840 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a97f61010504d15a334c2564939f5ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:38 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:40538
I20260923 17:34:38.629437 22979 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a97f61010504d15a334c2564939f5ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:38 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:59254
I20260923 17:34:38.630023 22701 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a97f61010504d15a334c2564939f5ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:38 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:43988
I20260923 17:34:38.651142 23141 tablet_replica.cc:330] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb: stopping tablet replica
I20260923 17:34:38.651656 23141 raft_consensus.cc:2229] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:34:38.652320 23139 tablet_replica.cc:330] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9: stopping tablet replica
I20260923 17:34:38.652767 23141 raft_consensus.cc:2258] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:38.653306 23139 raft_consensus.cc:2229] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:34:38.653848 23139 raft_consensus.cc:2258] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:38.654806 23141 ts_tablet_manager.cc:1905] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:38.656116 23139 ts_tablet_manager.cc:1905] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:38.657655 23138 tablet_replica.cc:330] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e: stopping tablet replica
I20260923 17:34:38.657866 23138 raft_consensus.cc:2229] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:34:38.658492 23138 raft_consensus.cc:2258] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:38.659631 23138 ts_tablet_manager.cc:1905] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:38.666085 23141 ts_tablet_manager.cc:1918] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:34:38.666620 23138 ts_tablet_manager.cc:1918] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:34:38.666785 23138 log.cc:1196] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/5a97f61010504d15a334c2564939f5ef
I20260923 17:34:38.671862 23139 ts_tablet_manager.cc:1918] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:34:38.672120 23139 log.cc:1196] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/5a97f61010504d15a334c2564939f5ef
I20260923 17:34:38.672463 23141 log.cc:1196] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/5a97f61010504d15a334c2564939f5ef
I20260923 17:34:38.673013 23139 ts_tablet_manager.cc:1939] T 5a97f61010504d15a334c2564939f5ef P f19426ade30947f081be4568f1ab29e9: Deleting consensus metadata
I20260923 17:34:38.673564 23141 ts_tablet_manager.cc:1939] T 5a97f61010504d15a334c2564939f5ef P 9908ba2c3f2f4b88a520c0ea8fa8edcb: Deleting consensus metadata
I20260923 17:34:38.674813 22550 catalog_manager.cc:4841] TS f19426ade30947f081be4568f1ab29e9 (127.20.240.67:36865): tablet 5a97f61010504d15a334c2564939f5ef (table test-table [id=019d0ecfa8fb42efbd76cf50a1ee7d90]) successfully deleted
I20260923 17:34:38.676239 22549 catalog_manager.cc:4841] TS 9908ba2c3f2f4b88a520c0ea8fa8edcb (127.20.240.65:39589): tablet 5a97f61010504d15a334c2564939f5ef (table test-table [id=019d0ecfa8fb42efbd76cf50a1ee7d90]) successfully deleted
I20260923 17:34:38.678519 23138 ts_tablet_manager.cc:1939] T 5a97f61010504d15a334c2564939f5ef P 2979bbaf9f264d79a6488d364451210e: Deleting consensus metadata
Sep 23 17:34:38 dist-test-slave-ppjl krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184877, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:34:38.679800 22549 catalog_manager.cc:4841] TS 2979bbaf9f264d79a6488d364451210e (127.20.240.66:36083): tablet 5a97f61010504d15a334c2564939f5ef (table test-table [id=019d0ecfa8fb42efbd76cf50a1ee7d90]) successfully deleted
I20260923 17:34:38.688879 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.653470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43994 (local address 127.20.240.65:39589)
0923 17:34:38.653648 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:38.653655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:38.667823 (+ 14168us) server_negotiation.cc:407] Connection header received
0923 17:34:38.668130 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:38.668136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:38.668222 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:38.668293 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:38.670749 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.671986 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.673945 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.674584 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.674680 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.680639 (+ 5959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:38.680655 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:38.680676 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:38.680745 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:38.685104 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:38.686212 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:38.686224 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:38.686228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:38.686409 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:38.686874 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:38.686879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:38.686882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:38.687411 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:38.687752 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:38.688062 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 17:34:38.688314 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260923 17:34:38.689777 22766 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43994
I20260923 17:34:38.729090 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.700281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.20.240.126:38733)
0923 17:34:38.712285 (+ 12004us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:38.712292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:38.712316 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:34:38.712387 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:38.712392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:38.712478 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:38.712611 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:38.714671 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.715960 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.717961 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.718657 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.718803 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.721989 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:38.721997 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:38.722002 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:38.722053 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:38.725647 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:38.726538 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:38.726544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:38.726549 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:38.726729 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:38.727304 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:38.727310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:38.727314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:38.727702 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:38.728027 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:38.728438 (+ 411us) server_negotiation.cc:299] Negotiation successful
0923 17:34:38.728694 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11849,"thread_start_us":11672,"threads_started":1}
W20260923 17:34:38.729764 22564 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37556
I20260923 17:34:38.733826 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 22635
I20260923 17:34:38.752609 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.734902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.20.240.65:39589)
0923 17:34:38.735085 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:38.735091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:38.735212 (+ 121us) server_negotiation.cc:407] Connection header received
0923 17:34:38.735489 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:38.735495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:38.735582 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:38.735682 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:38.737078 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.738213 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.740066 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.740781 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.740923 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.743891 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:38.743898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:38.743903 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:38.743958 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:38.748157 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:38.749295 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:38.749307 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:38.749311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:38.749510 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:38.749998 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:38.750003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:38.750007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:38.750388 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:38.750647 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:38.751943 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0923 17:34:38.752218 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 17:34:38.753505 22766 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44004
W20260923 17:34:38.754251 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 22635: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:34:38.754446 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 22635
I20260923 17:34:38.776208 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 22775
Sep 23 17:34:38 dist-test-slave-ppjl krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184877, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:34:38.791780 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.776965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40570 (local address 127.20.240.66:36083)
0923 17:34:38.777144 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:38.777150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:38.777232 (+ 82us) server_negotiation.cc:407] Connection header received
0923 17:34:38.777431 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:38.777437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:38.777526 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:38.777625 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:38.778819 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.779654 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.780940 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.781494 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.781613 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.785665 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:38.785683 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:38.785699 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:38.785738 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:38.788803 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:38.789538 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:38.789545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:38.789548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:38.789724 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:38.790116 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:38.790120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:38.790122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:38.790566 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:38.790856 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:38.791212 (+ 356us) server_negotiation.cc:299] Negotiation successful
0923 17:34:38.791454 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 17:34:38.792520 22905 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40570
W20260923 17:34:38.793292 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 22775: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:34:38.793457 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 22775
I20260923 17:34:38.808194 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 22914
Sep 23 17:34:38 dist-test-slave-ppjl krb5kdc[22519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184877, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:34:38.824258 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:38.808997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59274 (local address 127.20.240.67:36865)
0923 17:34:38.809189 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:38.809194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:38.809214 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:34:38.809281 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:38.809287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:38.809359 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:38.809449 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:38.810710 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.811462 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.812737 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:38.813187 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:38.813305 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:38.817626 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:38.817644 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:38.817661 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:38.817704 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:38.820800 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:38.821688 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:38.821696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:38.821698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:38.821855 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:38.822326 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:38.822330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:38.822332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:38.822756 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:38.823121 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:38.823592 (+ 471us) server_negotiation.cc:299] Negotiation successful
0923 17:34:38.823839 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 17:34:38.825006 23044 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59274
W20260923 17:34:38.825718 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 22914: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:34:38.825897 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 22914
I20260923 17:34:38.844017 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 22532
I20260923 17:34:38.845309 22604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:34:38.965096 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 22532
2026-09-23T17:34:38Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4890 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:34:39 dist-test-slave-ppjl krb5kdc[23150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:34:39 dist-test-slave-ppjl krb5kdc[23150](info): set up 2 sockets
Sep 23 17:34:39 dist-test-slave-ppjl krb5kdc[23150](info): commencing operation
krb5kdc: starting...
W20260923 17:34:39.979803 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:34:40 dist-test-slave-ppjl krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:34:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:34:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:34:40.166239 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:40443
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:40443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:40.489472 23163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:40.489789 23163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:40.489905 23163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:40.497293 23163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:34:40.497413 23163 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 17:34:40.497460 23163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:40.497524 23163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:40.497596 23163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:34:40.497671 23163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:34:40.508057 23163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:40443
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:40443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:40.509371 23163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:40.510694 23163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:40.522359 23168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:40.522477 23171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:40.522505 23169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:40.523530 23163 server_base.cc:1034] running on GCE node
I20260923 17:34:40.524407 23163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:40.525970 23163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:40.527256 23163 hybrid_clock.cc:648] HybridClock initialized: now 1790184880527180 us; error 35 us; skew 500 ppm
I20260923 17:34:40.527694 23163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:40 dist-test-slave-ppjl krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184880, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:40.531123 23163 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:34:40.532819 23163 webserver.cc:460] Webserver started at http://127.20.240.126:45909/ using document root <none> and password file <none>
I20260923 17:34:40.533531 23163 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:40.533635 23163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:40.533937 23163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:40.537011 23163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "4e9ab21fea16458eb413929177ed7820"
format_stamp: "Formatted at 2026-09-23 17:34:40 on dist-test-slave-ppjl"
server_key: "d03fc7c7d16d80f9d5009715b1e0aa3b"
server_key_iv: "f24fa63ad303ec594bc7e968fd72acdc"
server_key_version: "encryptionkey@0"
I20260923 17:34:40.537781 23163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "4e9ab21fea16458eb413929177ed7820"
format_stamp: "Formatted at 2026-09-23 17:34:40 on dist-test-slave-ppjl"
server_key: "d03fc7c7d16d80f9d5009715b1e0aa3b"
server_key_iv: "f24fa63ad303ec594bc7e968fd72acdc"
server_key_version: "encryptionkey@0"
I20260923 17:34:40.543570 23163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:34:40.547586 23178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:40.549125 23163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:34:40.549319 23163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "4e9ab21fea16458eb413929177ed7820"
format_stamp: "Formatted at 2026-09-23 17:34:40 on dist-test-slave-ppjl"
server_key: "d03fc7c7d16d80f9d5009715b1e0aa3b"
server_key_iv: "f24fa63ad303ec594bc7e968fd72acdc"
server_key_version: "encryptionkey@0"
I20260923 17:34:40.549541 23163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:40.573037 23163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:40.577769 23163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:40.578064 23163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:40.599677 23163 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:40443
I20260923 17:34:40.599726 23239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:40443 every 8 connection(s)
I20260923 17:34:40.601079 23163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:34:40.604482 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 23163
I20260923 17:34:40.604671 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:34:40.605255 23240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:40.605366 21441 external_mini_cluster.cc:1428] Setting key fa15ededfb47aad3ff2abd3f9bca8011
I20260923 17:34:40.618607 23240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820: Bootstrap starting.
Sep 23 17:34:40 dist-test-slave-ppjl krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:40.622398 23240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:40.623648 23240 log.cc:824] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:40.626562 23240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820: No bootstrap required, opened a new log
I20260923 17:34:40.628307 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:40.607594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.20.240.126:40443)
0923 17:34:40.608543 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:40.608556 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:40.608616 (+ 60us) server_negotiation.cc:407] Connection header received
0923 17:34:40.610219 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:40.610252 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:40.610711 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:40.611120 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:40.612665 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:40.613955 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:40.615285 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:40.616100 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:40.616318 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:40.620772 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:40.620782 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:40.620799 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:40.620847 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:40.624102 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:40.624880 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:40.624887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:40.624895 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:40.625078 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:40.625474 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:40.625477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:40.625480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:40.626018 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:40.626315 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:40.626756 (+ 441us) server_negotiation.cc:299] Negotiation successful
0923 17:34:40.627249 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":362,"threads_started":1}
I20260923 17:34:40.634527 23240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9ab21fea16458eb413929177ed7820" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40443 } }
I20260923 17:34:40.635036 23240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:40.635178 23240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9ab21fea16458eb413929177ed7820, State: Initialized, Role: FOLLOWER
I20260923 17:34:40.635885 23240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9ab21fea16458eb413929177ed7820" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40443 } }
I20260923 17:34:40.636113 23240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:34:40.636216 23240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:34:40.636350 23240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:40.638522 23240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9ab21fea16458eb413929177ed7820" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40443 } }
I20260923 17:34:40.639036 23240 leader_election.cc:302] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e9ab21fea16458eb413929177ed7820; no voters:
I20260923 17:34:40.639678 23240 leader_election.cc:288] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:34:40.639865 23245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:40.640748 23245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [term 1 LEADER]: Becoming Leader. State: Replica: 4e9ab21fea16458eb413929177ed7820, State: Running, Role: LEADER
I20260923 17:34:40.641340 23245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9ab21fea16458eb413929177ed7820" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40443 } }
I20260923 17:34:40.642005 23240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:34:40.644541 23247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e9ab21fea16458eb413929177ed7820. Latest consensus state: current_term: 1 leader_uuid: "4e9ab21fea16458eb413929177ed7820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9ab21fea16458eb413929177ed7820" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40443 } } }
I20260923 17:34:40.644588 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e9ab21fea16458eb413929177ed7820" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e9ab21fea16458eb413929177ed7820" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40443 } } }
I20260923 17:34:40.644981 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:40.644979 23247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820 [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:40.646623 23254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:34:40.652180 23254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:34:40.661170 23254 catalog_manager.cc:1347] Generated new cluster ID: 402a19f2fab04b3992304d2974f9faec
I20260923 17:34:40.661300 23254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:34:40.694236 23254 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:34:40.695765 23254 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:34:40.704228 23254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820: Generated new TSK 0
I20260923 17:34:40.705127 23254 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:34:40.760169 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:40443
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:41.090896 23266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:41.091246 23266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:41.091403 23266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:41.098765 23266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:41.098898 23266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:41.099025 23266 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 17:34:41.099092 23266 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:34:41.099220 23266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:34:41.109618 23266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.240.126:40443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:41.110992 23266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:41.112428 23266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:41.126215 23271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:41.126346 23272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:41.126251 23274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:41.127348 23266 server_base.cc:1034] running on GCE node
I20260923 17:34:41.128091 23266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:41.129132 23266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:41.130378 23266 hybrid_clock.cc:648] HybridClock initialized: now 1790184881130302 us; error 61 us; skew 500 ppm
I20260923 17:34:41.130752 23266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:41 dist-test-slave-ppjl krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184881, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:41.134110 23266 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:34:41.135802 23266 webserver.cc:460] Webserver started at http://127.20.240.65:40311/ using document root <none> and password file <none>
I20260923 17:34:41.136536 23266 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:41.136670 23266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:41.137002 23266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:41.140232 23266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "cc54d0b0eee54fa49d96f5e63e778294"
format_stamp: "Formatted at 2026-09-23 17:34:41 on dist-test-slave-ppjl"
server_key: "575ebe714d82b69324634cf579dd7e17"
server_key_iv: "83cf438fbba8c476a51d1f6d3291158f"
server_key_version: "encryptionkey@0"
I20260923 17:34:41.141073 23266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "cc54d0b0eee54fa49d96f5e63e778294"
format_stamp: "Formatted at 2026-09-23 17:34:41 on dist-test-slave-ppjl"
server_key: "575ebe714d82b69324634cf579dd7e17"
server_key_iv: "83cf438fbba8c476a51d1f6d3291158f"
server_key_version: "encryptionkey@0"
I20260923 17:34:41.146971 23266 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260923 17:34:41.150887 23281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:41.152364 23266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:34:41.152578 23266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "cc54d0b0eee54fa49d96f5e63e778294"
format_stamp: "Formatted at 2026-09-23 17:34:41 on dist-test-slave-ppjl"
server_key: "575ebe714d82b69324634cf579dd7e17"
server_key_iv: "83cf438fbba8c476a51d1f6d3291158f"
server_key_version: "encryptionkey@0"
I20260923 17:34:41.152856 23266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:41.175767 23266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:41.180390 23266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:41.180693 23266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:41.183462 23266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:41.183593 23266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:41.183710 23266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:41.183820 23266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:34:41 dist-test-slave-ppjl krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184881, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:41.209615 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:41.184320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:49305 (local address 127.20.240.126:40443)
0923 17:34:41.184791 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:41.184795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:41.186145 (+ 1350us) server_negotiation.cc:407] Connection header received
0923 17:34:41.188320 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:41.188328 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:41.188434 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:41.188561 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:41.190916 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.191911 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.195143 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.195871 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.196017 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:41.201730 (+ 5713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:41.201737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:41.201740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:41.201779 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:41.205402 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:41.206527 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:41.206533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:41.206536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:41.206689 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:41.207390 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:41.207395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:41.207399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:41.207672 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:41.207790 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:41.208971 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0923 17:34:41.209237 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":242,"threads_started":1}
I20260923 17:34:41.211485 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:41.185045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:40443 (local address 127.20.240.65:49305)
0923 17:34:41.185967 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:41.186011 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:41.187952 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:41.188927 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:41.188937 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:41.189744 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:41.190695 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:41.190724 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.192114 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.192121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:41.194841 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:41.194870 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.196422 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.196431 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:41.196809 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:41.197848 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:41.197900 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:41.201436 (+ 3536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:41.205582 (+ 4146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:41.205590 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:41.205611 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:41.206324 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:41.206857 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:41.206862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:41.206866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:41.207246 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:41.207808 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:41.207817 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:41.208414 (+ 597us) client_negotiation.cc:769] Sending connection context
0923 17:34:41.208851 (+ 437us) client_negotiation.cc:240] Negotiation successful
0923 17:34:41.209668 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":374,"threads_started":1}
I20260923 17:34:41.227178 23266 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:38891
I20260923 17:34:41.227192 23401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:38891 every 8 connection(s)
I20260923 17:34:41.228658 23266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:34:41.230159 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 23266
I20260923 17:34:41.230372 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:34:41.230983 23402 heartbeater.cc:347] Connected to a master server at 127.20.240.126:40443
I20260923 17:34:41.231180 21441 external_mini_cluster.cc:1428] Setting key 7d74945b67a89cb90e4966df53f7543d
I20260923 17:34:41.231326 23402 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:41.232194 23402 heartbeater.cc:511] Master 127.20.240.126:40443 requested a full tablet report, sending...
I20260923 17:34:41.233609 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:41.218276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:40443 (local address 127.20.240.65:41585)
0923 17:34:41.218986 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:41.219008 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:41.219199 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:41.219640 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:41.219645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:41.220307 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:41.220762 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:41.220778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.222094 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.222108 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:41.223875 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:41.223896 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.225016 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.225025 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:41.225220 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:41.225907 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:41.225930 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:41.226928 (+ 998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:41.230638 (+ 3710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:41.230644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:41.230648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:41.231235 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:41.231712 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:41.231717 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:41.231721 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:41.231893 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:41.232558 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:41.232563 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:41.232765 (+ 202us) client_negotiation.cc:769] Sending connection context
0923 17:34:41.232927 (+ 162us) client_negotiation.cc:240] Negotiation successful
0923 17:34:41.233227 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":438,"threads_started":1}
I20260923 17:34:41.234368 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:41.218207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:41585 (local address 127.20.240.126:40443)
0923 17:34:41.218393 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:41.218400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:41.219111 (+ 711us) server_negotiation.cc:407] Connection header received
0923 17:34:41.219423 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:41.219427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:41.219499 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:41.219587 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:41.220929 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.221930 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.224087 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.224617 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.224744 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:41.227129 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:41.227134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:41.227138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:41.227207 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:41.230499 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:41.231404 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:41.231410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:41.231413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:41.231581 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:41.232021 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:41.232026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:41.232029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:41.232429 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:41.232578 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:41.233935 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0923 17:34:41.234130 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:34:41.234839 23195 ts_manager.cc:194] Registered new tserver with Master: cc54d0b0eee54fa49d96f5e63e778294 (127.20.240.65:38891)
I20260923 17:34:41.236835 23195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:49305
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:34:41.275385 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:40443
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:41.597745 23406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:41.598061 23406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:41.598171 23406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:41.609239 23406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:41.609401 23406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:41.609542 23406 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 17:34:41.609624 23406 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:34:41.609728 23406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:34:41.624385 23406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.240.126:40443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:41.625667 23406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:41.627038 23406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:41.640211 23411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:41.640234 23414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:41.640164 23412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:41.641129 23406 server_base.cc:1034] running on GCE node
I20260923 17:34:41.641714 23406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:41.642783 23406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:41.643994 23406 hybrid_clock.cc:648] HybridClock initialized: now 1790184881643944 us; error 43 us; skew 500 ppm
I20260923 17:34:41.644316 23406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:41 dist-test-slave-ppjl krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184881, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:41.647678 23406 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:34:41.649318 23406 webserver.cc:460] Webserver started at http://127.20.240.66:44035/ using document root <none> and password file <none>
I20260923 17:34:41.650027 23406 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:41.650133 23406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:41.650470 23406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:41.653602 23406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "ca70020b320847939189b92deabb4935"
format_stamp: "Formatted at 2026-09-23 17:34:41 on dist-test-slave-ppjl"
server_key: "e0c756332fc1534c9c89a3b6aa4c9ad3"
server_key_iv: "31872e69be24f2171598c113f7bb0e2a"
server_key_version: "encryptionkey@0"
I20260923 17:34:41.654407 23406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "ca70020b320847939189b92deabb4935"
format_stamp: "Formatted at 2026-09-23 17:34:41 on dist-test-slave-ppjl"
server_key: "e0c756332fc1534c9c89a3b6aa4c9ad3"
server_key_iv: "31872e69be24f2171598c113f7bb0e2a"
server_key_version: "encryptionkey@0"
I20260923 17:34:41.660022 23406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:34:41.663857 23421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:41.665309 23406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:34:41.665539 23406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "ca70020b320847939189b92deabb4935"
format_stamp: "Formatted at 2026-09-23 17:34:41 on dist-test-slave-ppjl"
server_key: "e0c756332fc1534c9c89a3b6aa4c9ad3"
server_key_iv: "31872e69be24f2171598c113f7bb0e2a"
server_key_version: "encryptionkey@0"
I20260923 17:34:41.665730 23406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:41.683218 23406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:41.687633 23406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:41.687918 23406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:41.690774 23406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:41.690900 23406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:41.691021 23406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:41.691110 23406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:34:41 dist-test-slave-ppjl krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184881, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:41.715934 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:41.691509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:43999 (local address 127.20.240.126:40443)
0923 17:34:41.691716 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:41.691723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:41.693158 (+ 1435us) server_negotiation.cc:407] Connection header received
0923 17:34:41.695470 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:41.695476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:41.695566 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:41.695657 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:41.698005 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.698900 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.701412 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.702027 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.702194 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:41.707699 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:41.707708 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:41.707714 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:41.707768 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:41.711338 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:41.712684 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:41.712692 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:41.712697 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:41.712895 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:41.713634 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:41.713642 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:41.713645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:41.714106 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:41.714274 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:41.715297 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0923 17:34:41.715589 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 17:34:41.717095 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:41.692107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:40443 (local address 127.20.240.66:43999)
0923 17:34:41.692970 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:41.693019 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:41.695086 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:41.695962 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:41.695979 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:41.696820 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:41.697776 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:41.697799 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.699135 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.699139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:41.701229 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:41.701248 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.702323 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.702334 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:41.702699 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:41.703783 (+ 1084us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:41.703880 (+ 97us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:41.707368 (+ 3488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:41.711596 (+ 4228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:41.711607 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:41.711630 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:41.712420 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:41.713038 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:41.713044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:41.713048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:41.713463 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:41.714249 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:41.714258 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:41.714818 (+ 560us) client_negotiation.cc:769] Sending connection context
0923 17:34:41.715127 (+ 309us) client_negotiation.cc:240] Negotiation successful
0923 17:34:41.715778 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":363,"threads_started":1}
I20260923 17:34:41.741379 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:41.726332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:42427 (local address 127.20.240.126:40443)
0923 17:34:41.726555 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:41.726562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:41.727147 (+ 585us) server_negotiation.cc:407] Connection header received
0923 17:34:41.727402 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:41.727406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:41.727477 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:41.727564 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:41.728949 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.729701 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.732023 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.732540 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.732691 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:41.735110 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:41.735117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:41.735120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:41.735182 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:41.737943 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:41.738797 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:41.738803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:41.738807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:41.738998 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:41.739551 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:41.739556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:41.739561 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:41.740090 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:41.740253 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:41.740907 (+ 654us) server_negotiation.cc:299] Negotiation successful
0923 17:34:41.741140 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 17:34:41.741701 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:41.726172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:40443 (local address 127.20.240.66:42427)
0923 17:34:41.726993 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:41.727021 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:41.727254 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:41.727623 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:41.727628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:41.728302 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:41.728754 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:41.728769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.729842 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.729847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:41.731863 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:41.731883 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:41.732973 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:41.732985 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:41.733174 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:41.733872 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:41.733899 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:41.734842 (+ 943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:41.738086 (+ 3244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:41.738091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:41.738096 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:41.738661 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:41.739189 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:41.739196 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:41.739200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:41.739387 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:41.740326 (+ 939us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:41.740332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:41.740573 (+ 241us) client_negotiation.cc:769] Sending connection context
0923 17:34:41.740771 (+ 198us) client_negotiation.cc:240] Negotiation successful
0923 17:34:41.741210 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":361,"threads_started":1}
I20260923 17:34:41.747849 23406 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:34489
I20260923 17:34:41.747885 23540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:34489 every 8 connection(s)
I20260923 17:34:41.749212 23406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:34:41.751492 23541 heartbeater.cc:347] Connected to a master server at 127.20.240.126:40443
I20260923 17:34:41.751703 23541 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:41.752375 23541 heartbeater.cc:511] Master 127.20.240.126:40443 requested a full tablet report, sending...
I20260923 17:34:41.753913 23195 ts_manager.cc:194] Registered new tserver with Master: ca70020b320847939189b92deabb4935 (127.20.240.66:34489)
I20260923 17:34:41.754093 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 23406
I20260923 17:34:41.754251 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:34:41.754739 21441 external_mini_cluster.cc:1428] Setting key caed7c1905eb7966b6a3899c8066b0f9
I20260923 17:34:41.754967 23195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:43999
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:34:41.795804 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:40443
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:42.124686 23545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:42.124991 23545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:42.125141 23545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:42.133193 23545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:42.133337 23545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:42.133469 23545 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 17:34:42.133538 23545 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:34:42.133641 23545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:34:42.146193 23545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.240.126:40443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:42.147737 23545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:42.149130 23545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:42.162789 23551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:42.162865 23550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:42.162842 23553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:42.163658 23545 server_base.cc:1034] running on GCE node
I20260923 17:34:42.164259 23545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:42.165215 23545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:42.166417 23545 hybrid_clock.cc:648] HybridClock initialized: now 1790184882166351 us; error 61 us; skew 500 ppm
I20260923 17:34:42.166728 23545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:42 dist-test-slave-ppjl krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184882, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:42.174556 23545 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:34:42.176266 23545 webserver.cc:460] Webserver started at http://127.20.240.67:35507/ using document root <none> and password file <none>
I20260923 17:34:42.177006 23545 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:42.177135 23545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:42.177448 23545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:42.180704 23545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "757ca798cea6407eb879293c28b026ba"
format_stamp: "Formatted at 2026-09-23 17:34:42 on dist-test-slave-ppjl"
server_key: "0b48ea6f9b8d9c8bab2b99a0521400d4"
server_key_iv: "19d9131fbb234962ed705c0810a602a4"
server_key_version: "encryptionkey@0"
I20260923 17:34:42.181756 23545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "757ca798cea6407eb879293c28b026ba"
format_stamp: "Formatted at 2026-09-23 17:34:42 on dist-test-slave-ppjl"
server_key: "0b48ea6f9b8d9c8bab2b99a0521400d4"
server_key_iv: "19d9131fbb234962ed705c0810a602a4"
server_key_version: "encryptionkey@0"
I20260923 17:34:42.187744 23545 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260923 17:34:42.191622 23560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:42.193235 23545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:34:42.193430 23545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "757ca798cea6407eb879293c28b026ba"
format_stamp: "Formatted at 2026-09-23 17:34:42 on dist-test-slave-ppjl"
server_key: "0b48ea6f9b8d9c8bab2b99a0521400d4"
server_key_iv: "19d9131fbb234962ed705c0810a602a4"
server_key_version: "encryptionkey@0"
I20260923 17:34:42.193648 23545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:42.206892 23545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:42.211511 23545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:42.211843 23545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:42.214591 23545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:42.214695 23545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:42.214833 23545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:42.214903 23545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:34:42 dist-test-slave-ppjl krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184882, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:42.240407 23402 heartbeater.cc:503] Master 127.20.240.126:40443 was elected leader, sending a full tablet report...
I20260923 17:34:42.240661 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.215407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:34709 (local address 127.20.240.126:40443)
0923 17:34:42.215605 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:42.215611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:42.217084 (+ 1473us) server_negotiation.cc:407] Connection header received
0923 17:34:42.219620 (+ 2536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:42.219627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:42.219735 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:42.219891 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:42.222330 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.223544 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.225272 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.225884 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.226021 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.231952 (+ 5931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:42.231961 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:42.231966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:42.232016 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:42.236051 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:42.237317 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:42.237324 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:42.237327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:42.237516 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:42.238170 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:42.238175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:42.238178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:42.238575 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:42.238733 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:42.240087 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0923 17:34:42.240367 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:34:42.242410 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.215974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:40443 (local address 127.20.240.67:34709)
0923 17:34:42.216876 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:42.216924 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:42.219200 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:42.220187 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:42.220202 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:42.221088 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:42.222043 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.222068 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.223734 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.223738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.225109 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.225123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.226161 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.226171 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.226520 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.227427 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:42.227463 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:42.231595 (+ 4132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:42.236267 (+ 4672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:42.236279 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:42.236301 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:42.237161 (+ 860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:42.237678 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:42.237689 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:42.237693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:42.238036 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:42.238735 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:42.238746 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:42.239397 (+ 651us) client_negotiation.cc:769] Sending connection context
0923 17:34:42.239883 (+ 486us) client_negotiation.cc:240] Negotiation successful
0923 17:34:42.240560 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":366,"threads_started":1}
I20260923 17:34:42.255545 23545 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:38305
I20260923 17:34:42.255548 23679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:38305 every 8 connection(s)
I20260923 17:34:42.257335 23545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:34:42.259742 23680 heartbeater.cc:347] Connected to a master server at 127.20.240.126:40443
I20260923 17:34:42.259972 23680 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:42.260716 23680 heartbeater.cc:511] Master 127.20.240.126:40443 requested a full tablet report, sending...
I20260923 17:34:42.262284 23195 ts_manager.cc:194] Registered new tserver with Master: 757ca798cea6407eb879293c28b026ba (127.20.240.67:38305)
I20260923 17:34:42.263435 23195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:34709
I20260923 17:34:42.264076 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 23545
I20260923 17:34:42.264256 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:34:42.264825 21441 external_mini_cluster.cc:1428] Setting key 2162c045b1a7b6a18101b38a783e2afe
I20260923 17:34:42.266263 23668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.250537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:40443 (local address 127.20.240.67:44433)
0923 17:34:42.251746 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:42.251773 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:42.251977 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:42.252468 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:42.252475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:42.253200 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:42.253645 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.253660 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.255236 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.255243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.256994 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.257010 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.257883 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.257891 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.258132 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.258827 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:42.258849 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:42.259626 (+ 777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:42.263263 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:42.263268 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:42.263271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:42.263898 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:42.264425 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:42.264432 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:42.264436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:42.264574 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:42.265169 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:42.265173 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:42.265349 (+ 176us) client_negotiation.cc:769] Sending connection context
0923 17:34:42.265506 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 17:34:42.265856 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":844,"threads_started":1}
I20260923 17:34:42.266294 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.250446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:44433 (local address 127.20.240.126:40443)
0923 17:34:42.250644 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:42.250650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:42.251908 (+ 1258us) server_negotiation.cc:407] Connection header received
0923 17:34:42.252142 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:42.252148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:42.252276 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:42.252406 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:42.253849 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.254986 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.257158 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.257660 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.257779 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.259840 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:42.259846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:42.259850 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:42.259893 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:42.263061 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:42.264060 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:42.264064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:42.264066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:42.264233 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:42.264686 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:42.264690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:42.264693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:42.264980 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:42.265117 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:42.265643 (+ 526us) server_negotiation.cc:299] Negotiation successful
0923 17:34:42.265897 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:34:42.268195 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:34:42 dist-test-slave-ppjl krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:34:42 dist-test-slave-ppjl krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:42.314867 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.300020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.20.240.126:40443)
0923 17:34:42.300185 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:42.300190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:42.300463 (+ 273us) server_negotiation.cc:407] Connection header received
0923 17:34:42.300620 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:42.300625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:42.300698 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:42.300793 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:42.302305 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.303106 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.304517 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.304988 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.305099 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.309145 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:42.309151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:42.309155 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:42.309197 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:42.311990 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:42.312728 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:42.312736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:42.312739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:42.312908 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:42.313406 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:42.313412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:42.313415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:42.313740 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:42.313951 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:42.314321 (+ 370us) server_negotiation.cc:299] Negotiation successful
0923 17:34:42.314549 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:34:42.319648 23194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:34:42.326805 23194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:34:42.353816 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.340184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:34489 (local address 127.0.0.1:40750)
0923 17:34:42.341174 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:42.341231 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:42.341446 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:42.343200 (+ 1754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:42.343208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:42.343242 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:42.343692 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.343705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.346743 (+ 3038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.346749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.350441 (+ 3692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.350457 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.352680 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.352691 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.352927 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.352994 (+ 67us) client_negotiation.cc:769] Sending connection context
0923 17:34:42.353121 (+ 127us) client_negotiation.cc:240] Negotiation successful
0923 17:34:42.353357 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":4,"thread_start_us":2583,"threads_started":1}
I20260923 17:34:42.354616 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.340395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.20.240.66:34489)
0923 17:34:42.341992 (+ 1597us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:42.342495 (+ 503us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:42.342526 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:34:42.342638 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:42.342646 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:42.342922 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:42.343906 (+ 984us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:42.344126 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.346596 (+ 2470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.350644 (+ 4048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.352154 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.352295 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.352529 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:42.354108 (+ 1579us) server_negotiation.cc:299] Negotiation successful
0923 17:34:42.354321 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1452,"thread_start_us":1318,"threads_started":1}
I20260923 17:34:42.355012 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.340855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50358 (local address 127.20.240.65:38891)
0923 17:34:42.342366 (+ 1511us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:42.342375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:42.343230 (+ 855us) server_negotiation.cc:407] Connection header received
0923 17:34:42.343580 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:42.343588 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:42.343823 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:42.343995 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:42.345346 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.346891 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.350839 (+ 3948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.352260 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.352371 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.352529 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:42.354523 (+ 1994us) server_negotiation.cc:299] Negotiation successful
0923 17:34:42.354715 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1364,"thread_start_us":1174,"threads_started":1}
I20260923 17:34:42.355806 23693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.340402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:38891 (local address 127.0.0.1:50358)
0923 17:34:42.343107 (+ 2705us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:42.343129 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:42.343382 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:42.344223 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:42.344228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:42.344263 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:42.345156 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.345169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.347134 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.347143 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.350666 (+ 3523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.350678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.352739 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.352744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.354272 (+ 1528us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.354308 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:34:42.355238 (+ 930us) client_negotiation.cc:240] Negotiation successful
0923 17:34:42.355402 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2562,"mutex_wait_us":1102,"thread_start_us":668,"threads_started":1}
I20260923 17:34:42.358873 23471 tablet_service.cc:1467] Processing CreateTablet for tablet ec0bd078d6cc42e1948f508a6edb9662 (DEFAULT_TABLE table=test-table [id=609b683796bb45a4ad8038fc28d53b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:42.359480 23332 tablet_service.cc:1467] Processing CreateTablet for tablet ec0bd078d6cc42e1948f508a6edb9662 (DEFAULT_TABLE table=test-table [id=609b683796bb45a4ad8038fc28d53b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:42.360456 23471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec0bd078d6cc42e1948f508a6edb9662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:42.360772 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.343116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:38305 (local address 127.0.0.1:35552)
0923 17:34:42.349844 (+ 6728us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:42.349870 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:42.350020 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:42.351433 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:42.351439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:42.351471 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:42.351857 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.351871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.354552 (+ 2681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.354557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.357862 (+ 3305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.357882 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.360006 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.360011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.360172 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.360206 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:34:42.360288 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:34:42.360435 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6588,"thread_start_us":7188,"threads_started":1}
I20260923 17:34:42.360844 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.343457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.20.240.67:38305)
0923 17:34:42.349495 (+ 6038us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:42.349504 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:42.350171 (+ 667us) server_negotiation.cc:407] Connection header received
0923 17:34:42.350362 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:42.350371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:42.350664 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:42.351060 (+ 396us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:42.352118 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.354348 (+ 2230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.358113 (+ 3765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.359568 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.359710 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.359896 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:42.360380 (+ 484us) server_negotiation.cc:299] Negotiation successful
0923 17:34:42.360556 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5846,"thread_start_us":5648,"threads_started":1}
I20260923 17:34:42.360842 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec0bd078d6cc42e1948f508a6edb9662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:42.363875 23605 tablet_service.cc:1467] Processing CreateTablet for tablet ec0bd078d6cc42e1948f508a6edb9662 (DEFAULT_TABLE table=test-table [id=609b683796bb45a4ad8038fc28d53b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:42.365247 23605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec0bd078d6cc42e1948f508a6edb9662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:42.376468 23699 tablet_bootstrap.cc:492] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935: Bootstrap starting.
I20260923 17:34:42.375910 23700 tablet_bootstrap.cc:492] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294: Bootstrap starting.
I20260923 17:34:42.377269 23701 tablet_bootstrap.cc:492] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba: Bootstrap starting.
I20260923 17:34:42.380092 23700 tablet_bootstrap.cc:654] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:42.380180 23699 tablet_bootstrap.cc:654] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:42.380725 23701 tablet_bootstrap.cc:654] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:42.381508 23699 log.cc:824] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:42.381899 23701 log.cc:824] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:42.381920 23700 log.cc:824] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:42.384948 23700 tablet_bootstrap.cc:492] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294: No bootstrap required, opened a new log
I20260923 17:34:42.385069 23699 tablet_bootstrap.cc:492] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935: No bootstrap required, opened a new log
I20260923 17:34:42.385092 23701 tablet_bootstrap.cc:492] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba: No bootstrap required, opened a new log
I20260923 17:34:42.385251 23700 ts_tablet_manager.cc:1397] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260923 17:34:42.385352 23701 ts_tablet_manager.cc:1397] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 17:34:42.385365 23699 ts_tablet_manager.cc:1397] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 17:34:42.392676 23700 raft_consensus.cc:348] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.393391 23700 raft_consensus.cc:374] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:42.393507 23700 raft_consensus.cc:729] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc54d0b0eee54fa49d96f5e63e778294, State: Initialized, Role: FOLLOWER
I20260923 17:34:42.394155 23700 consensus_queue.cc:260] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.396162 23700 ts_tablet_manager.cc:1428] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260923 17:34:42.396167 23699 raft_consensus.cc:348] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.396173 23701 raft_consensus.cc:348] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.396698 23699 raft_consensus.cc:374] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:42.396687 23701 raft_consensus.cc:374] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:42.396806 23699 raft_consensus.cc:729] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca70020b320847939189b92deabb4935, State: Initialized, Role: FOLLOWER
I20260923 17:34:42.396801 23701 raft_consensus.cc:729] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757ca798cea6407eb879293c28b026ba, State: Initialized, Role: FOLLOWER
I20260923 17:34:42.397491 23699 consensus_queue.cc:260] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.397491 23701 consensus_queue.cc:260] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.398697 23680 heartbeater.cc:503] Master 127.20.240.126:40443 was elected leader, sending a full tablet report...
I20260923 17:34:42.399631 23701 ts_tablet_manager.cc:1428] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20260923 17:34:42.401024 23541 heartbeater.cc:503] Master 127.20.240.126:40443 was elected leader, sending a full tablet report...
I20260923 17:34:42.401810 23699 ts_tablet_manager.cc:1428] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935: Time spent starting tablet: real 0.016s user 0.014s sys 0.003s
W20260923 17:34:42.483383 23403 tablet.cc:2369] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:34:42.501314 23542 tablet.cc:2369] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:34:42.509164 23681 tablet.cc:2369] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:34:42.528258 23705 raft_consensus.cc:482] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:34:42.528630 23705 raft_consensus.cc:504] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.530514 23705 leader_election.cc:288] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca70020b320847939189b92deabb4935 (127.20.240.66:34489), 757ca798cea6407eb879293c28b026ba (127.20.240.67:38305)
I20260923 17:34:42.537513 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.531178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:46257 (local address 127.20.240.67:38305)
0923 17:34:42.531435 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:42.531441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:42.531665 (+ 224us) server_negotiation.cc:407] Connection header received
0923 17:34:42.531881 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:42.531886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:42.531973 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:42.532079 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:42.532693 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.533725 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.535676 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.536581 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.536709 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.536763 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:42.537128 (+ 365us) server_negotiation.cc:299] Negotiation successful
0923 17:34:42.537260 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:34:42.537545 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.530825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:38305 (local address 127.20.240.65:46257)
0923 17:34:42.531559 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:42.531578 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:42.531752 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:42.532111 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:42.532119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:42.532149 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:42.532546 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.532559 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.533852 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.533858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.535529 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.535541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.536790 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.536797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.536967 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.537001 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:34:42.537102 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 17:34:42.537243 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":335,"threads_started":1}
I20260923 17:34:42.537989 23709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.531647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:34489 (local address 127.20.240.65:44693)
0923 17:34:42.532290 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:42.532304 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:42.532416 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:42.532707 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:42.532712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:42.532734 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:42.533100 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.533112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.534335 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.534340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.535936 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.535951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.537224 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.537230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.537405 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.537462 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 17:34:42.537555 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 17:34:42.537674 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":1,"thread_start_us":351,"threads_started":1}
I20260923 17:34:42.538161 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.531029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:44693 (local address 127.20.240.66:34489)
0923 17:34:42.531276 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:42.531282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:42.532400 (+ 1118us) server_negotiation.cc:407] Connection header received
0923 17:34:42.532522 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:42.532527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:42.532601 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:42.532685 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:42.533299 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.534215 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.536085 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.537022 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.537170 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.537238 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:42.537584 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 17:34:42.537927 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:34:42.538676 23630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec0bd078d6cc42e1948f508a6edb9662" candidate_uuid: "cc54d0b0eee54fa49d96f5e63e778294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757ca798cea6407eb879293c28b026ba" is_pre_election: true
I20260923 17:34:42.538930 23495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec0bd078d6cc42e1948f508a6edb9662" candidate_uuid: "cc54d0b0eee54fa49d96f5e63e778294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca70020b320847939189b92deabb4935" is_pre_election: true
I20260923 17:34:42.539201 23630 raft_consensus.cc:2454] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc54d0b0eee54fa49d96f5e63e778294 in term 0.
I20260923 17:34:42.539407 23495 raft_consensus.cc:2454] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc54d0b0eee54fa49d96f5e63e778294 in term 0.
I20260923 17:34:42.539837 23285 leader_election.cc:302] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757ca798cea6407eb879293c28b026ba, cc54d0b0eee54fa49d96f5e63e778294; no voters:
I20260923 17:34:42.540344 23705 raft_consensus.cc:2781] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:34:42.540495 23705 raft_consensus.cc:482] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:34:42.540606 23705 raft_consensus.cc:3037] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:42.543076 23705 raft_consensus.cc:504] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.543781 23705 leader_election.cc:288] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [CANDIDATE]: Term 1 election: Requested vote from peers ca70020b320847939189b92deabb4935 (127.20.240.66:34489), 757ca798cea6407eb879293c28b026ba (127.20.240.67:38305)
I20260923 17:34:42.544279 23630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec0bd078d6cc42e1948f508a6edb9662" candidate_uuid: "cc54d0b0eee54fa49d96f5e63e778294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757ca798cea6407eb879293c28b026ba"
I20260923 17:34:42.544344 23495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec0bd078d6cc42e1948f508a6edb9662" candidate_uuid: "cc54d0b0eee54fa49d96f5e63e778294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca70020b320847939189b92deabb4935"
I20260923 17:34:42.544539 23630 raft_consensus.cc:3037] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:42.544553 23495 raft_consensus.cc:3037] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:42.546965 23630 raft_consensus.cc:2454] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc54d0b0eee54fa49d96f5e63e778294 in term 1.
I20260923 17:34:42.546985 23495 raft_consensus.cc:2454] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc54d0b0eee54fa49d96f5e63e778294 in term 1.
I20260923 17:34:42.547426 23285 leader_election.cc:302] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757ca798cea6407eb879293c28b026ba, cc54d0b0eee54fa49d96f5e63e778294; no voters:
I20260923 17:34:42.547734 23705 raft_consensus.cc:2781] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:42.548262 23705 raft_consensus.cc:686] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [term 1 LEADER]: Becoming Leader. State: Replica: cc54d0b0eee54fa49d96f5e63e778294, State: Running, Role: LEADER
I20260923 17:34:42.548786 23705 consensus_queue.cc:237] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.553696 23194 catalog_manager.cc:5495] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc54d0b0eee54fa49d96f5e63e778294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "cc54d0b0eee54fa49d96f5e63e778294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } health_report { overall_health: UNKNOWN } } }
Sep 23 17:34:42 dist-test-slave-ppjl krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184880, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:42.624558 23717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.610181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:40443 (local address 127.0.0.1:60272)
0923 17:34:42.610903 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:42.610922 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:42.611070 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:42.611441 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:42.611447 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:42.612387 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:42.612739 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.612751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.613858 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.613864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.615125 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.615137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.615977 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.615982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.616110 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.616692 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:42.616708 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:42.618888 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:42.621780 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:42.621793 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:42.621800 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:42.622303 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:42.622773 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:42.622778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:42.622781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:42.622945 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:42.623624 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:42.623631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:42.623798 (+ 167us) client_negotiation.cc:769] Sending connection context
0923 17:34:42.623944 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 17:34:42.624177 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":400,"threads_started":1}
I20260923 17:34:42.624624 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.610056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60272 (local address 127.20.240.126:40443)
0923 17:34:42.610255 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:42.610261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:42.611023 (+ 762us) server_negotiation.cc:407] Connection header received
0923 17:34:42.611206 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:42.611212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:42.611301 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:42.611386 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:42.612913 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.613688 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.615337 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.615812 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.615924 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.619068 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:42.619073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:42.619077 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:42.619107 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:42.621643 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:42.622483 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:42.622487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:42.622489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:42.622630 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:42.623051 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:42.623055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:42.623057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:42.623492 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:42.623609 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:42.624050 (+ 441us) server_negotiation.cc:299] Negotiation successful
0923 17:34:42.624314 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":65}
I20260923 17:34:42.629803 23194 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:60272:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 17:34:42.630970 23194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:34:42.638288 23471 tablet_service.cc:1467] Processing CreateTablet for tablet a5b98c6f0a5f41639e52f58c18d841e9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c66efbfd43a411f9f63f3b2d3b30479]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:34:42.638288 23332 tablet_service.cc:1467] Processing CreateTablet for tablet a5b98c6f0a5f41639e52f58c18d841e9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c66efbfd43a411f9f63f3b2d3b30479]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:34:42.638545 23605 tablet_service.cc:1467] Processing CreateTablet for tablet a5b98c6f0a5f41639e52f58c18d841e9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c66efbfd43a411f9f63f3b2d3b30479]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:34:42.639341 23605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b98c6f0a5f41639e52f58c18d841e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:42.639354 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b98c6f0a5f41639e52f58c18d841e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:42.639361 23471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5b98c6f0a5f41639e52f58c18d841e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:42.645496 23701 tablet_bootstrap.cc:492] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba: Bootstrap starting.
I20260923 17:34:42.646987 23701 tablet_bootstrap.cc:654] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:42.647012 23699 tablet_bootstrap.cc:492] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935: Bootstrap starting.
I20260923 17:34:42.646996 23700 tablet_bootstrap.cc:492] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294: Bootstrap starting.
I20260923 17:34:42.648806 23699 tablet_bootstrap.cc:654] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:42.648856 23700 tablet_bootstrap.cc:654] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:42.649796 23701 tablet_bootstrap.cc:492] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba: No bootstrap required, opened a new log
I20260923 17:34:42.649999 23701 ts_tablet_manager.cc:1397] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:34:42.650923 23699 tablet_bootstrap.cc:492] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935: No bootstrap required, opened a new log
I20260923 17:34:42.650806 23701 raft_consensus.cc:348] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.651021 23701 raft_consensus.cc:374] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:42.651062 23699 ts_tablet_manager.cc:1397] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260923 17:34:42.651117 23701 raft_consensus.cc:729] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757ca798cea6407eb879293c28b026ba, State: Initialized, Role: FOLLOWER
I20260923 17:34:42.651429 23701 consensus_queue.cc:260] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.651863 23699 raft_consensus.cc:348] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.652064 23699 raft_consensus.cc:374] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:42.652153 23699 raft_consensus.cc:729] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca70020b320847939189b92deabb4935, State: Initialized, Role: FOLLOWER
I20260923 17:34:42.651877 23706 tablet_replica.cc:439] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } } }
I20260923 17:34:42.652287 23706 tablet_replica.cc:442] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:34:42.652406 23699 consensus_queue.cc:260] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.652675 23700 tablet_bootstrap.cc:492] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294: No bootstrap required, opened a new log
I20260923 17:34:42.652825 23701 ts_tablet_manager.cc:1428] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:34:42.652859 23700 ts_tablet_manager.cc:1397] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260923 17:34:42.652870 23699 ts_tablet_manager.cc:1428] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:34:42.653275 23707 tablet_replica.cc:439] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } } }
I20260923 17:34:42.653702 23707 tablet_replica.cc:442] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:34:42.653967 23700 raft_consensus.cc:348] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.654225 23700 raft_consensus.cc:374] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:42.654328 23700 raft_consensus.cc:729] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc54d0b0eee54fa49d96f5e63e778294, State: Initialized, Role: FOLLOWER
I20260923 17:34:42.654599 23700 consensus_queue.cc:260] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.655267 23700 ts_tablet_manager.cc:1428] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:34:42.655383 23705 tablet_replica.cc:439] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } } }
I20260923 17:34:42.655668 23705 tablet_replica.cc:442] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:34:42.839274 23707 raft_consensus.cc:482] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:34:42.839560 23707 raft_consensus.cc:504] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.841081 23707 leader_election.cc:288] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc54d0b0eee54fa49d96f5e63e778294 (127.20.240.65:38891), 757ca798cea6407eb879293c28b026ba (127.20.240.67:38305)
I20260923 17:34:42.850407 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.841513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:36031 (local address 127.20.240.65:38891)
0923 17:34:42.841719 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:42.841724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:42.842448 (+ 724us) server_negotiation.cc:407] Connection header received
0923 17:34:42.842678 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:42.842684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:42.842761 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:42.842846 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:42.843643 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.844776 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.847973 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.849241 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.849430 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.849509 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:42.849989 (+ 480us) server_negotiation.cc:299] Negotiation successful
0923 17:34:42.850134 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:34:42.850449 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.841513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:57055 (local address 127.20.240.67:38305)
0923 17:34:42.841711 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:42.841717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:42.842387 (+ 670us) server_negotiation.cc:407] Connection header received
0923 17:34:42.842586 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:42.842591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:42.842676 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:42.842832 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:42.843655 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.844797 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.847965 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.849318 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.849452 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.849515 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:42.850003 (+ 488us) server_negotiation.cc:299] Negotiation successful
0923 17:34:42.850136 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 17:34:42.850627 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.841379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:38305 (local address 127.20.240.66:57055)
0923 17:34:42.842219 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:42.842238 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:42.842421 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:42.842849 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:42.842853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:42.842881 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:42.843439 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.843452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.844949 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.844954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.847762 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.847782 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.849529 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.849534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.849755 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.849823 (+ 68us) client_negotiation.cc:769] Sending connection context
0923 17:34:42.849934 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 17:34:42.850143 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":240,"thread_start_us":510,"threads_started":1}
I20260923 17:34:42.851003 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.841450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:38891 (local address 127.20.240.66:36031)
0923 17:34:42.842325 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:42.842339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:42.842536 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:42.842888 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:42.842891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:42.842908 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:42.843439 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.843452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.844948 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.844954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.847762 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.847782 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.849516 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.849524 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.849760 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.849820 (+ 60us) client_negotiation.cc:769] Sending connection context
0923 17:34:42.849922 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 17:34:42.850143 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":407,"thread_start_us":491,"threads_started":1}
I20260923 17:34:42.851387 23630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b98c6f0a5f41639e52f58c18d841e9" candidate_uuid: "ca70020b320847939189b92deabb4935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757ca798cea6407eb879293c28b026ba" is_pre_election: true
I20260923 17:34:42.851642 23630 raft_consensus.cc:2454] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca70020b320847939189b92deabb4935 in term 0.
I20260923 17:34:42.852012 23352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b98c6f0a5f41639e52f58c18d841e9" candidate_uuid: "ca70020b320847939189b92deabb4935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc54d0b0eee54fa49d96f5e63e778294" is_pre_election: true
I20260923 17:34:42.852124 23425 leader_election.cc:302] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757ca798cea6407eb879293c28b026ba, ca70020b320847939189b92deabb4935; no voters:
I20260923 17:34:42.852409 23352 raft_consensus.cc:2454] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca70020b320847939189b92deabb4935 in term 0.
I20260923 17:34:42.852586 23707 raft_consensus.cc:2781] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:34:42.852706 23707 raft_consensus.cc:482] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:34:42.852805 23707 raft_consensus.cc:3037] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:42.855127 23707 raft_consensus.cc:504] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.855754 23707 leader_election.cc:288] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [CANDIDATE]: Term 1 election: Requested vote from peers cc54d0b0eee54fa49d96f5e63e778294 (127.20.240.65:38891), 757ca798cea6407eb879293c28b026ba (127.20.240.67:38305)
I20260923 17:34:42.856271 23630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b98c6f0a5f41639e52f58c18d841e9" candidate_uuid: "ca70020b320847939189b92deabb4935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757ca798cea6407eb879293c28b026ba"
I20260923 17:34:42.856251 23352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5b98c6f0a5f41639e52f58c18d841e9" candidate_uuid: "ca70020b320847939189b92deabb4935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc54d0b0eee54fa49d96f5e63e778294"
I20260923 17:34:42.856489 23630 raft_consensus.cc:3037] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:42.856508 23352 raft_consensus.cc:3037] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:42.858544 23630 raft_consensus.cc:2454] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca70020b320847939189b92deabb4935 in term 1.
I20260923 17:34:42.858747 23352 raft_consensus.cc:2454] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca70020b320847939189b92deabb4935 in term 1.
I20260923 17:34:42.858930 23425 leader_election.cc:302] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757ca798cea6407eb879293c28b026ba, ca70020b320847939189b92deabb4935; no voters:
I20260923 17:34:42.859274 23707 raft_consensus.cc:2781] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:42.859675 23707 raft_consensus.cc:686] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [term 1 LEADER]: Becoming Leader. State: Replica: ca70020b320847939189b92deabb4935, State: Running, Role: LEADER
I20260923 17:34:42.860204 23707 consensus_queue.cc:237] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:42.863777 23723 tablet_replica.cc:439] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935: TxnStatusTablet state changed. Reason: New leader ca70020b320847939189b92deabb4935. Latest consensus state: current_term: 1 leader_uuid: "ca70020b320847939189b92deabb4935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } } }
I20260923 17:34:42.864189 23723 tablet_replica.cc:442] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:34:42.864821 23725 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 17:34:42.865054 23725 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 17:34:42.865305 23193 catalog_manager.cc:5495] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca70020b320847939189b92deabb4935 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "ca70020b320847939189b92deabb4935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } health_report { overall_health: UNKNOWN } } }
I20260923 17:34:42.892904 23717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.887403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:34489 (local address 127.0.0.1:40764)
0923 17:34:42.887649 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:42.887668 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:42.887846 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:42.888289 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:42.888293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:42.888306 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:34:42.888485 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.888494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.889548 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.889554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.890539 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.890551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.891293 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.891300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.891457 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.891517 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:34:42.892356 (+ 839us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:34:42.892380 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:34:42.892460 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 17:34:42.892617 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 17:34:42.892928 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.887412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.20.240.66:34489)
0923 17:34:42.887611 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:42.887621 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:42.887737 (+ 116us) server_negotiation.cc:407] Connection header received
0923 17:34:42.888051 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:42.888055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:42.888133 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:42.888251 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:34:42.888610 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.889423 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.890681 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.891101 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.891228 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.891689 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:34:42.892240 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:34:42.892384 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:42.892495 (+ 111us) server_negotiation.cc:299] Negotiation successful
0923 17:34:42.892581 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:34:42.901850 23352 raft_consensus.cc:1261] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Refusing update from remote peer ca70020b320847939189b92deabb4935: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:34:42.901938 23630 raft_consensus.cc:1261] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Refusing update from remote peer ca70020b320847939189b92deabb4935: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:34:42.902848 23707 consensus_queue.cc:1035] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:34:42.903203 23723 consensus_queue.cc:1035] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:34:42.910553 23705 tablet_replica.cc:439] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294: TxnStatusTablet state changed. Reason: New leader ca70020b320847939189b92deabb4935. Latest consensus state: current_term: 1 leader_uuid: "ca70020b320847939189b92deabb4935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } } }
I20260923 17:34:42.910925 23705 tablet_replica.cc:442] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:34:42.913807 23728 mvcc.cc:205] Tried to move back new op lower bound from 7332597280355766272 to 7332597280199245824. Current Snapshot: MvccSnapshot[applied={T|T < 7332597280355766272}]
I20260923 17:34:42.917351 23705 tablet_replica.cc:439] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca70020b320847939189b92deabb4935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } } }
I20260923 17:34:42.917657 23705 tablet_replica.cc:442] T a5b98c6f0a5f41639e52f58c18d841e9 P cc54d0b0eee54fa49d96f5e63e778294: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:34:42.924433 23707 tablet_replica.cc:439] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca70020b320847939189b92deabb4935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } } }
I20260923 17:34:42.924782 23707 tablet_replica.cc:442] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:34:42.928570 23706 tablet_replica.cc:439] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba: TxnStatusTablet state changed. Reason: New leader ca70020b320847939189b92deabb4935. Latest consensus state: current_term: 1 leader_uuid: "ca70020b320847939189b92deabb4935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } } }
I20260923 17:34:42.928901 23706 tablet_replica.cc:442] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:34:42.931246 23707 tablet_replica.cc:439] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca70020b320847939189b92deabb4935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } } }
I20260923 17:34:42.931552 23707 tablet_replica.cc:442] T a5b98c6f0a5f41639e52f58c18d841e9 P ca70020b320847939189b92deabb4935: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:34:42.938390 23706 tablet_replica.cc:439] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca70020b320847939189b92deabb4935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } } }
I20260923 17:34:42.938689 23706 tablet_replica.cc:442] T a5b98c6f0a5f41639e52f58c18d841e9 P 757ca798cea6407eb879293c28b026ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:34:42.946558 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.938251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.20.240.65:38891)
0923 17:34:42.938559 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:42.938565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:42.940815 (+ 2250us) server_negotiation.cc:407] Connection header received
0923 17:34:42.941092 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:42.941096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:42.941186 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:42.941333 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:34:42.941856 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.942699 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.944255 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.944725 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.944862 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.945323 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:34:42.945930 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:34:42.946097 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:42.946160 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 17:34:42.946248 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 17:34:42.965670 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.958945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:47719 (local address 127.20.240.66:34489)
0923 17:34:42.959197 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:42.959205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:42.959676 (+ 471us) server_negotiation.cc:407] Connection header received
0923 17:34:42.959863 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:42.959867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:42.959938 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:42.960019 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:34:42.960608 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.961456 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.963583 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.964109 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.964255 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.964841 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:34:42.964973 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:34:42.965069 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:42.965266 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 17:34:42.965363 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:34:42.965822 23740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:42.958906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:34489 (local address 127.20.240.65:47719)
0923 17:34:42.959566 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:42.959587 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:42.959746 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:42.960048 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:42.960053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:42.960065 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:34:42.960417 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.960430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.961588 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.961594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.963405 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:42.963424 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:42.964326 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:42.964333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:42.964526 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:42.964605 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:34:42.965079 (+ 474us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:34:42.965091 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 17:34:42.965212 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 17:34:42.965378 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":337,"threads_started":1}
I20260923 17:34:42.965726 23705 consensus_queue.cc:1035] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:34:42.977782 23495 raft_consensus.cc:1261] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Refusing update from remote peer cc54d0b0eee54fa49d96f5e63e778294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:34:42.979202 23710 consensus_queue.cc:1035] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:34:43.011819 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:43.005572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:48445 (local address 127.20.240.65:38891)
0923 17:34:43.005735 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:43.005740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:43.006360 (+ 620us) server_negotiation.cc:407] Connection header received
0923 17:34:43.006614 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:43.006619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:43.006704 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:43.006765 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:34:43.007576 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.008390 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.009932 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.010383 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.010489 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:43.011027 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:34:43.011200 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:34:43.011261 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:43.011441 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 17:34:43.011523 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 17:34:43.012084 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:43.005656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:38891 (local address 127.20.240.66:48445)
0923 17:34:43.006255 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:43.006269 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:43.006423 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:43.006957 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:43.006962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:43.006973 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:34:43.007409 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:43.007420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.008520 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.008525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:43.009787 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:43.009803 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.010562 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.010568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:43.010722 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:43.010802 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:34:43.011323 (+ 521us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:34:43.011330 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:34:43.011399 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 17:34:43.011539 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":781,"threads_started":1}
I20260923 17:34:43.057588 23195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:34:43.057957 23195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:34:43.062426 23195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820: Sending DeleteTablet for 3 replicas of tablet ec0bd078d6cc42e1948f508a6edb9662
I20260923 17:34:43.063890 23605 tablet_service.cc:1514] Processing DeleteTablet for tablet ec0bd078d6cc42e1948f508a6edb9662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:43 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:35552
I20260923 17:34:43.063901 23471 tablet_service.cc:1514] Processing DeleteTablet for tablet ec0bd078d6cc42e1948f508a6edb9662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:43 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:40750
I20260923 17:34:43.065115 23332 tablet_service.cc:1514] Processing DeleteTablet for tablet ec0bd078d6cc42e1948f508a6edb9662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:43 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:50358
I20260923 17:34:43.072045 23751 tablet_replica.cc:330] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294: stopping tablet replica
I20260923 17:34:43.073239 23751 raft_consensus.cc:2229] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:34:43.073314 23195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:34:43.074122 23195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:34:43.074887 23751 raft_consensus.cc:2258] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:43.083204 23470 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1f82c5c8f64cd9b03541a82e3f4ea9 (DEFAULT_TABLE table=test-table [id=68b757e67c37444086f9eda8336bc076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:43.083822 23470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1f82c5c8f64cd9b03541a82e3f4ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:43.082603 23332 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1f82c5c8f64cd9b03541a82e3f4ea9 (DEFAULT_TABLE table=test-table [id=68b757e67c37444086f9eda8336bc076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:43.084491 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1f82c5c8f64cd9b03541a82e3f4ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:43.085306 23749 tablet_replica.cc:330] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935: stopping tablet replica
I20260923 17:34:43.087368 23749 raft_consensus.cc:2229] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:34:43.087776 23749 raft_consensus.cc:2258] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:43.088124 23751 ts_tablet_manager.cc:1905] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:43.090862 23610 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1f82c5c8f64cd9b03541a82e3f4ea9 (DEFAULT_TABLE table=test-table [id=68b757e67c37444086f9eda8336bc076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:43.091470 23610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1f82c5c8f64cd9b03541a82e3f4ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:43.092110 23750 tablet_replica.cc:330] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba: stopping tablet replica
I20260923 17:34:43.094012 23750 raft_consensus.cc:2229] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:34:43.094395 23750 raft_consensus.cc:2258] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:43.094820 23749 ts_tablet_manager.cc:1905] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:43.095144 23700 tablet_bootstrap.cc:492] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294: Bootstrap starting.
I20260923 17:34:43.096964 23700 tablet_bootstrap.cc:654] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:43.097811 23750 ts_tablet_manager.cc:1905] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:43.098069 23701 tablet_bootstrap.cc:492] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba: Bootstrap starting.
I20260923 17:34:43.101552 23701 tablet_bootstrap.cc:654] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:43.098239 23699 tablet_bootstrap.cc:492] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935: Bootstrap starting.
I20260923 17:34:43.104287 23699 tablet_bootstrap.cc:654] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:43.108779 23700 tablet_bootstrap.cc:492] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294: No bootstrap required, opened a new log
I20260923 17:34:43.108983 23700 ts_tablet_manager.cc:1397] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260923 17:34:43.109860 23700 raft_consensus.cc:348] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:43.110131 23700 raft_consensus.cc:374] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:43.110214 23700 raft_consensus.cc:729] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc54d0b0eee54fa49d96f5e63e778294, State: Initialized, Role: FOLLOWER
I20260923 17:34:43.110514 23700 consensus_queue.cc:260] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:43.111150 23700 ts_tablet_manager.cc:1428] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:34:43.115773 23751 ts_tablet_manager.cc:1918] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:34:43.118311 23749 ts_tablet_manager.cc:1918] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:34:43.118643 23751 log.cc:1196] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/ec0bd078d6cc42e1948f508a6edb9662
I20260923 17:34:43.119547 23751 ts_tablet_manager.cc:1939] T ec0bd078d6cc42e1948f508a6edb9662 P cc54d0b0eee54fa49d96f5e63e778294: Deleting consensus metadata
I20260923 17:34:43.121279 23701 tablet_bootstrap.cc:492] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba: No bootstrap required, opened a new log
I20260923 17:34:43.121465 23701 ts_tablet_manager.cc:1397] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.003s
I20260923 17:34:43.121587 23179 catalog_manager.cc:4841] TS cc54d0b0eee54fa49d96f5e63e778294 (127.20.240.65:38891): tablet ec0bd078d6cc42e1948f508a6edb9662 (table test-table [id=609b683796bb45a4ad8038fc28d53b2a]) successfully deleted
I20260923 17:34:43.121778 23749 log.cc:1196] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/ec0bd078d6cc42e1948f508a6edb9662
I20260923 17:34:43.122105 23750 ts_tablet_manager.cc:1918] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:34:43.122284 23750 log.cc:1196] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/ec0bd078d6cc42e1948f508a6edb9662
I20260923 17:34:43.122325 23701 raft_consensus.cc:348] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:43.122593 23701 raft_consensus.cc:374] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:43.122679 23701 raft_consensus.cc:729] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757ca798cea6407eb879293c28b026ba, State: Initialized, Role: FOLLOWER
I20260923 17:34:43.122848 23750 ts_tablet_manager.cc:1939] T ec0bd078d6cc42e1948f508a6edb9662 P 757ca798cea6407eb879293c28b026ba: Deleting consensus metadata
I20260923 17:34:43.122942 23701 consensus_queue.cc:260] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:43.123509 23701 ts_tablet_manager.cc:1428] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 17:34:43.124620 23749 ts_tablet_manager.cc:1939] T ec0bd078d6cc42e1948f508a6edb9662 P ca70020b320847939189b92deabb4935: Deleting consensus metadata
I20260923 17:34:43.125408 23699 tablet_bootstrap.cc:492] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935: No bootstrap required, opened a new log
I20260923 17:34:43.125633 23699 ts_tablet_manager.cc:1397] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.011s
I20260923 17:34:43.127260 23699 raft_consensus.cc:348] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:43.127540 23699 raft_consensus.cc:374] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:43.127661 23699 raft_consensus.cc:729] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca70020b320847939189b92deabb4935, State: Initialized, Role: FOLLOWER
I20260923 17:34:43.128031 23699 consensus_queue.cc:260] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:43.129055 23182 catalog_manager.cc:4841] TS 757ca798cea6407eb879293c28b026ba (127.20.240.67:38305): tablet ec0bd078d6cc42e1948f508a6edb9662 (table test-table [id=609b683796bb45a4ad8038fc28d53b2a]) successfully deleted
I20260923 17:34:43.129783 23699 ts_tablet_manager.cc:1428] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260923 17:34:43.131523 23180 catalog_manager.cc:4841] TS ca70020b320847939189b92deabb4935 (127.20.240.66:34489): tablet ec0bd078d6cc42e1948f508a6edb9662 (table test-table [id=609b683796bb45a4ad8038fc28d53b2a]) successfully deleted
I20260923 17:34:43.230545 23706 raft_consensus.cc:482] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:34:43.230827 23706 raft_consensus.cc:504] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:43.232688 23706 leader_election.cc:288] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc54d0b0eee54fa49d96f5e63e778294 (127.20.240.65:38891), ca70020b320847939189b92deabb4935 (127.20.240.66:34489)
I20260923 17:34:43.256256 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:43.233684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:49353 (local address 127.20.240.65:38891)
0923 17:34:43.233894 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:43.233900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:43.242995 (+ 9095us) server_negotiation.cc:407] Connection header received
0923 17:34:43.243358 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:43.243364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:43.243470 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:43.243539 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:43.244408 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.245600 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.253327 (+ 7727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.254756 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.254952 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:43.255043 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:43.255759 (+ 716us) server_negotiation.cc:299] Negotiation successful
0923 17:34:43.255936 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 17:34:43.257491 23756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:43.233292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:38891 (local address 127.20.240.67:49353)
0923 17:34:43.242838 (+ 9546us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:43.242864 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:43.243138 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:43.243784 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:43.243790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:43.243831 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:43.244187 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:43.244202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.245840 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.245846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:43.249288 (+ 3442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:43.249307 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.255318 (+ 6011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.255325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:43.255522 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:43.255557 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:34:43.256853 (+ 1296us) client_negotiation.cc:240] Negotiation successful
0923 17:34:43.257106 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9366,"thread_start_us":15212,"threads_started":1}
I20260923 17:34:43.260066 23352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1f82c5c8f64cd9b03541a82e3f4ea9" candidate_uuid: "757ca798cea6407eb879293c28b026ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc54d0b0eee54fa49d96f5e63e778294" is_pre_election: true
I20260923 17:34:43.260365 23352 raft_consensus.cc:2454] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 757ca798cea6407eb879293c28b026ba in term 0.
I20260923 17:34:43.261500 23561 leader_election.cc:302] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757ca798cea6407eb879293c28b026ba, cc54d0b0eee54fa49d96f5e63e778294; no voters:
I20260923 17:34:43.262042 23706 raft_consensus.cc:2781] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:34:43.262166 23706 raft_consensus.cc:482] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:34:43.262246 23706 raft_consensus.cc:3037] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:43.265466 23706 raft_consensus.cc:504] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:43.267091 23352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1f82c5c8f64cd9b03541a82e3f4ea9" candidate_uuid: "757ca798cea6407eb879293c28b026ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc54d0b0eee54fa49d96f5e63e778294"
I20260923 17:34:43.267354 23352 raft_consensus.cc:3037] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:43.270668 23352 raft_consensus.cc:2454] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 757ca798cea6407eb879293c28b026ba in term 1.
I20260923 17:34:43.271430 23561 leader_election.cc:302] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757ca798cea6407eb879293c28b026ba, cc54d0b0eee54fa49d96f5e63e778294; no voters:
I20260923 17:34:43.271934 23706 leader_election.cc:288] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [CANDIDATE]: Term 1 election: Requested vote from peers cc54d0b0eee54fa49d96f5e63e778294 (127.20.240.65:38891), ca70020b320847939189b92deabb4935 (127.20.240.66:34489)
I20260923 17:34:43.272183 23706 raft_consensus.cc:2781] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:43.272379 23706 raft_consensus.cc:686] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [term 1 LEADER]: Becoming Leader. State: Replica: 757ca798cea6407eb879293c28b026ba, State: Running, Role: LEADER
I20260923 17:34:43.272847 23706 consensus_queue.cc:237] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:43.285058 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:43.234678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:52963 (local address 127.20.240.66:34489)
0923 17:34:43.234888 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:43.234894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:43.248896 (+ 14002us) server_negotiation.cc:407] Connection header received
0923 17:34:43.249192 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:43.249198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:43.249277 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:43.249339 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:34:43.250104 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.251365 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.282221 (+ 30856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.283637 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.283858 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:43.283935 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:43.284556 (+ 621us) server_negotiation.cc:299] Negotiation successful
0923 17:34:43.284729 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 17:34:43.286121 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:43.234262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:34489 (local address 127.20.240.67:52963)
0923 17:34:43.248780 (+ 14518us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:43.248802 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:43.249019 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:43.249528 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:43.249533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:43.249561 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:34:43.249920 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:43.249931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.251873 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.251880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:43.281955 (+ 30075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:43.281982 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.284123 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.284131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:43.284338 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:43.284375 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 17:34:43.285595 (+ 1220us) client_negotiation.cc:240] Negotiation successful
0923 17:34:43.285764 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14061,"thread_start_us":8304,"threads_started":1}
I20260923 17:34:43.287400 23495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1f82c5c8f64cd9b03541a82e3f4ea9" candidate_uuid: "757ca798cea6407eb879293c28b026ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca70020b320847939189b92deabb4935" is_pre_election: true
I20260923 17:34:43.287698 23495 raft_consensus.cc:2454] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 757ca798cea6407eb879293c28b026ba in term 0.
I20260923 17:34:43.288048 23494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1f82c5c8f64cd9b03541a82e3f4ea9" candidate_uuid: "757ca798cea6407eb879293c28b026ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca70020b320847939189b92deabb4935"
I20260923 17:34:43.288290 23494 raft_consensus.cc:3037] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:43.292771 23494 raft_consensus.cc:2454] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 757ca798cea6407eb879293c28b026ba in term 1.
I20260923 17:34:43.304534 23195 catalog_manager.cc:5495] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 757ca798cea6407eb879293c28b026ba (127.20.240.67). New cstate: current_term: 1 leader_uuid: "757ca798cea6407eb879293c28b026ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } health_report { overall_health: HEALTHY } } }
I20260923 17:34:43.360960 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:43.353788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.20.240.67:38305)
0923 17:34:43.354249 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:43.354256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:43.354274 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:34:43.354375 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:43.354380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:43.354587 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:43.354772 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:34:43.355206 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.356266 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.358160 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.358837 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.358981 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:43.359499 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:34:43.360214 (+ 715us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:34:43.360359 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:43.360492 (+ 133us) server_negotiation.cc:299] Negotiation successful
0923 17:34:43.360581 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":217,"threads_started":1}
I20260923 17:34:43.378897 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:43.372643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:34013 (local address 127.20.240.66:34489)
0923 17:34:43.372802 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:43.372806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:43.373499 (+ 693us) server_negotiation.cc:407] Connection header received
0923 17:34:43.373663 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:43.373669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:43.373792 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:43.373896 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:34:43.374414 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.375511 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.376913 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.377547 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.377746 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:43.378131 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:34:43.378293 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:34:43.378393 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:43.378520 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 17:34:43.378608 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:34:43.378924 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:43.372674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:34489 (local address 127.20.240.67:34013)
0923 17:34:43.373373 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:43.373391 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:43.373541 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:43.373952 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:43.373956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:43.373967 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:34:43.374254 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:43.374263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.375651 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.375655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:43.376765 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:43.376779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.377744 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.377750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:43.377879 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:43.377950 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:34:43.378389 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:34:43.378416 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:34:43.378476 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 17:34:43.378582 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":345,"threads_started":1}
I20260923 17:34:43.388124 23352 raft_consensus.cc:1261] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Refusing update from remote peer 757ca798cea6407eb879293c28b026ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:34:43.388211 23494 raft_consensus.cc:1261] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Refusing update from remote peer 757ca798cea6407eb879293c28b026ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:34:43.388779 23757 consensus_queue.cc:1035] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:34:43.389182 23706 consensus_queue.cc:1035] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:34:43.394582 23730 mvcc.cc:205] Tried to move back new op lower bound from 7332597282351800320 to 7332597281916715008. Current Snapshot: MvccSnapshot[applied={T|T < 7332597282351800320}]
I20260923 17:34:43.395471 23729 mvcc.cc:205] Tried to move back new op lower bound from 7332597282351800320 to 7332597281916715008. Current Snapshot: MvccSnapshot[applied={T|T < 7332597282351800320}]
I20260923 17:34:43.417364 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:43.410321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:38305 (local address 127.20.240.66:44761)
0923 17:34:43.410731 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:43.410751 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:43.410858 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:43.411657 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:43.411662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:43.411675 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:34:43.411882 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:43.411895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.413219 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.413225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:43.414758 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:43.414775 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.415983 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.415989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:43.416177 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:43.416322 (+ 145us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:34:43.416753 (+ 431us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:34:43.416766 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 17:34:43.416851 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 17:34:43.417016 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274}
I20260923 17:34:43.417927 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:43.411040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:44761 (local address 127.20.240.67:38305)
0923 17:34:43.411224 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:43.411230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:43.411248 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:34:43.411309 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:43.411314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:43.411397 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:43.411487 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:34:43.412041 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.413060 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.414937 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:43.415593 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:43.415811 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:43.416492 (+ 681us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:34:43.416642 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:34:43.416752 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:43.417597 (+ 845us) server_negotiation.cc:299] Negotiation successful
0923 17:34:43.417695 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260923 17:34:43.419085 23730 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 17:34:45.017254 23195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:34:45.017491 23195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:34:45.022006 23195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820: Sending DeleteTablet for 3 replicas of tablet 4b1f82c5c8f64cd9b03541a82e3f4ea9
I20260923 17:34:45.023022 23471 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b1f82c5c8f64cd9b03541a82e3f4ea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:45 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:40750
I20260923 17:34:45.023069 23332 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b1f82c5c8f64cd9b03541a82e3f4ea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:45 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:50358
I20260923 17:34:45.023073 23610 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b1f82c5c8f64cd9b03541a82e3f4ea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:45 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:35552
I20260923 17:34:45.035468 23792 tablet_replica.cc:330] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba: stopping tablet replica
I20260923 17:34:45.035688 23792 raft_consensus.cc:2229] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:34:45.036224 23792 raft_consensus.cc:2258] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:45.037510 23792 ts_tablet_manager.cc:1905] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:45.038781 23791 tablet_replica.cc:330] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294: stopping tablet replica
I20260923 17:34:45.039108 23791 raft_consensus.cc:2229] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:34:45.039659 23791 raft_consensus.cc:2258] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:45.039695 23790 tablet_replica.cc:330] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935: stopping tablet replica
I20260923 17:34:45.039891 23790 raft_consensus.cc:2229] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:34:45.040134 23790 raft_consensus.cc:2258] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:45.040979 23791 ts_tablet_manager.cc:1905] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:45.041062 23790 ts_tablet_manager.cc:1905] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:45.044780 23792 ts_tablet_manager.cc:1918] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:34:45.045011 23792 log.cc:1196] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/4b1f82c5c8f64cd9b03541a82e3f4ea9
I20260923 17:34:45.045603 23792 ts_tablet_manager.cc:1939] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P 757ca798cea6407eb879293c28b026ba: Deleting consensus metadata
I20260923 17:34:45.047724 23790 ts_tablet_manager.cc:1918] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:34:45.047859 23790 log.cc:1196] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/4b1f82c5c8f64cd9b03541a82e3f4ea9
I20260923 17:34:45.047981 23791 ts_tablet_manager.cc:1918] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:34:45.048120 23791 log.cc:1196] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/4b1f82c5c8f64cd9b03541a82e3f4ea9
I20260923 17:34:45.048300 23790 ts_tablet_manager.cc:1939] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P ca70020b320847939189b92deabb4935: Deleting consensus metadata
I20260923 17:34:45.048534 23791 ts_tablet_manager.cc:1939] T 4b1f82c5c8f64cd9b03541a82e3f4ea9 P cc54d0b0eee54fa49d96f5e63e778294: Deleting consensus metadata
I20260923 17:34:45.049486 23180 catalog_manager.cc:4841] TS ca70020b320847939189b92deabb4935 (127.20.240.66:34489): tablet 4b1f82c5c8f64cd9b03541a82e3f4ea9 (table test-table [id=68b757e67c37444086f9eda8336bc076]) successfully deleted
I20260923 17:34:45.049692 23182 catalog_manager.cc:4841] TS 757ca798cea6407eb879293c28b026ba (127.20.240.67:38305): tablet 4b1f82c5c8f64cd9b03541a82e3f4ea9 (table test-table [id=68b757e67c37444086f9eda8336bc076]) successfully deleted
I20260923 17:34:45.049754 23179 catalog_manager.cc:4841] TS cc54d0b0eee54fa49d96f5e63e778294 (127.20.240.65:38891): tablet 4b1f82c5c8f64cd9b03541a82e3f4ea9 (table test-table [id=68b757e67c37444086f9eda8336bc076]) successfully deleted
Sep 23 17:34:45 dist-test-slave-ppjl krb5kdc[23150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:34:45 dist-test-slave-ppjl krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:45.083619 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:45.067987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.20.240.126:40443)
0923 17:34:45.068603 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:45.068608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:45.068630 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:34:45.068688 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:45.068692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:45.068754 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:45.068897 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:45.070369 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:45.071256 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:45.072650 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:45.073108 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:45.073213 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:45.077610 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:45.077617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:45.077620 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:45.077671 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:45.080848 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:45.081619 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:45.081624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:45.081627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:45.081768 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:45.082139 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:45.082143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:45.082145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:45.082452 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:45.082684 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:45.083016 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 17:34:45.083285 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":267,"threads_started":1}
I20260923 17:34:45.087296 23195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:34:45.088011 23195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:34:45.095115 23332 tablet_service.cc:1467] Processing CreateTablet for tablet 5767ae77f6214c038014baf583554e40 (DEFAULT_TABLE table=test-table [id=c327a583ec694aa4b420a6ba9708f066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:45.095219 23471 tablet_service.cc:1467] Processing CreateTablet for tablet 5767ae77f6214c038014baf583554e40 (DEFAULT_TABLE table=test-table [id=c327a583ec694aa4b420a6ba9708f066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:45.095364 23610 tablet_service.cc:1467] Processing CreateTablet for tablet 5767ae77f6214c038014baf583554e40 (DEFAULT_TABLE table=test-table [id=c327a583ec694aa4b420a6ba9708f066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:45.095593 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5767ae77f6214c038014baf583554e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:45.095736 23471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5767ae77f6214c038014baf583554e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:45.095753 23610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5767ae77f6214c038014baf583554e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:45.101928 23801 tablet_bootstrap.cc:492] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294: Bootstrap starting.
I20260923 17:34:45.102699 23803 tablet_bootstrap.cc:492] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba: Bootstrap starting.
I20260923 17:34:45.103324 23802 tablet_bootstrap.cc:492] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935: Bootstrap starting.
I20260923 17:34:45.103870 23801 tablet_bootstrap.cc:654] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:45.105018 23802 tablet_bootstrap.cc:654] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:45.106182 23801 tablet_bootstrap.cc:492] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294: No bootstrap required, opened a new log
I20260923 17:34:45.106345 23801 ts_tablet_manager.cc:1397] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 17:34:45.106751 23803 tablet_bootstrap.cc:654] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:45.107295 23802 tablet_bootstrap.cc:492] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935: No bootstrap required, opened a new log
I20260923 17:34:45.107103 23801 raft_consensus.cc:348] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.107385 23801 raft_consensus.cc:374] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:45.107462 23802 ts_tablet_manager.cc:1397] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260923 17:34:45.107493 23801 raft_consensus.cc:729] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc54d0b0eee54fa49d96f5e63e778294, State: Initialized, Role: FOLLOWER
I20260923 17:34:45.107728 23801 consensus_queue.cc:260] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.108237 23802 raft_consensus.cc:348] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.108608 23802 raft_consensus.cc:374] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:45.108747 23802 raft_consensus.cc:729] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca70020b320847939189b92deabb4935, State: Initialized, Role: FOLLOWER
I20260923 17:34:45.108999 23803 tablet_bootstrap.cc:492] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba: No bootstrap required, opened a new log
I20260923 17:34:45.109140 23803 ts_tablet_manager.cc:1397] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260923 17:34:45.109053 23802 consensus_queue.cc:260] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.109634 23802 ts_tablet_manager.cc:1428] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:34:45.109973 23801 ts_tablet_manager.cc:1428] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:34:45.109920 23803 raft_consensus.cc:348] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.110144 23803 raft_consensus.cc:374] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:45.110231 23803 raft_consensus.cc:729] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757ca798cea6407eb879293c28b026ba, State: Initialized, Role: FOLLOWER
I20260923 17:34:45.110464 23803 consensus_queue.cc:260] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.111002 23803 ts_tablet_manager.cc:1428] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:34:45.212342 23785 raft_consensus.cc:482] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:34:45.212569 23785 raft_consensus.cc:504] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.213285 23785 leader_election.cc:288] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc54d0b0eee54fa49d96f5e63e778294 (127.20.240.65:38891), 757ca798cea6407eb879293c28b026ba (127.20.240.67:38305)
I20260923 17:34:45.214087 23351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5767ae77f6214c038014baf583554e40" candidate_uuid: "ca70020b320847939189b92deabb4935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc54d0b0eee54fa49d96f5e63e778294" is_pre_election: true
I20260923 17:34:45.214080 23629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5767ae77f6214c038014baf583554e40" candidate_uuid: "ca70020b320847939189b92deabb4935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757ca798cea6407eb879293c28b026ba" is_pre_election: true
I20260923 17:34:45.214392 23351 raft_consensus.cc:2454] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca70020b320847939189b92deabb4935 in term 0.
I20260923 17:34:45.214421 23629 raft_consensus.cc:2454] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca70020b320847939189b92deabb4935 in term 0.
I20260923 17:34:45.215009 23422 leader_election.cc:302] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 757ca798cea6407eb879293c28b026ba, ca70020b320847939189b92deabb4935, cc54d0b0eee54fa49d96f5e63e778294; no voters:
I20260923 17:34:45.215440 23785 raft_consensus.cc:2781] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:34:45.215560 23785 raft_consensus.cc:482] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:34:45.215651 23785 raft_consensus.cc:3037] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:45.217778 23785 raft_consensus.cc:504] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.218392 23785 leader_election.cc:288] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [CANDIDATE]: Term 1 election: Requested vote from peers cc54d0b0eee54fa49d96f5e63e778294 (127.20.240.65:38891), 757ca798cea6407eb879293c28b026ba (127.20.240.67:38305)
I20260923 17:34:45.218999 23351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5767ae77f6214c038014baf583554e40" candidate_uuid: "ca70020b320847939189b92deabb4935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc54d0b0eee54fa49d96f5e63e778294"
I20260923 17:34:45.219125 23629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5767ae77f6214c038014baf583554e40" candidate_uuid: "ca70020b320847939189b92deabb4935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757ca798cea6407eb879293c28b026ba"
I20260923 17:34:45.219264 23351 raft_consensus.cc:3037] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:45.219390 23629 raft_consensus.cc:3037] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:45.221495 23351 raft_consensus.cc:2454] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca70020b320847939189b92deabb4935 in term 1.
I20260923 17:34:45.221680 23629 raft_consensus.cc:2454] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca70020b320847939189b92deabb4935 in term 1.
I20260923 17:34:45.222045 23422 leader_election.cc:302] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca70020b320847939189b92deabb4935, cc54d0b0eee54fa49d96f5e63e778294; no voters:
I20260923 17:34:45.222404 23785 raft_consensus.cc:2781] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:45.222561 23785 raft_consensus.cc:686] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [term 1 LEADER]: Becoming Leader. State: Replica: ca70020b320847939189b92deabb4935, State: Running, Role: LEADER
I20260923 17:34:45.222873 23785 consensus_queue.cc:237] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.225601 23193 catalog_manager.cc:5495] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca70020b320847939189b92deabb4935 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "ca70020b320847939189b92deabb4935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } health_report { overall_health: UNKNOWN } } }
I20260923 17:34:45.272181 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:45.266485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40774 (local address 127.20.240.66:34489)
0923 17:34:45.267139 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:45.267144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:45.267188 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:34:45.267250 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:45.267253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:45.267325 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:45.267402 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:34:45.267759 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:45.268587 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:45.270462 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:45.270900 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:45.270991 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:45.271517 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:34:45.271631 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:34:45.271699 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:45.271798 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 17:34:45.271869 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":407,"threads_started":1}
I20260923 17:34:45.286629 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:45.281644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:41631 (local address 127.20.240.66:34489)
0923 17:34:45.281837 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:45.281844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:45.282170 (+ 326us) server_negotiation.cc:407] Connection header received
0923 17:34:45.282334 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:45.282345 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:45.282407 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:45.282475 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:34:45.282957 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:45.283672 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:45.284985 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:45.285484 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:45.285619 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:45.285951 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:34:45.286052 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:34:45.286121 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:45.286241 (+ 120us) server_negotiation.cc:299] Negotiation successful
0923 17:34:45.286310 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:34:45.286686 23813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:45.281448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:34489 (local address 127.20.240.66:41631)
0923 17:34:45.282079 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:45.282093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:45.282226 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:45.282505 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:45.282509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:45.282519 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:34:45.282755 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:45.282763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:45.283797 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:45.283802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:45.284817 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:45.284828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:45.285652 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:45.285658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:45.285780 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:45.285816 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:34:45.286142 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:34:45.286152 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 17:34:45.286216 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 17:34:45.286356 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":399,"threads_started":1}
I20260923 17:34:45.295236 23351 raft_consensus.cc:1261] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Refusing update from remote peer ca70020b320847939189b92deabb4935: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:34:45.295236 23629 raft_consensus.cc:1261] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Refusing update from remote peer ca70020b320847939189b92deabb4935: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:34:45.295801 23777 consensus_queue.cc:1035] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:34:45.296155 23785 consensus_queue.cc:1035] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:34:45.347972 23193 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:34:45.348304 23193 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:34:45.351727 23193 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820: Sending DeleteTablet for 3 replicas of tablet 5767ae77f6214c038014baf583554e40
I20260923 17:34:45.352692 23610 tablet_service.cc:1514] Processing DeleteTablet for tablet 5767ae77f6214c038014baf583554e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:45 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:35552
I20260923 17:34:45.352999 23792 tablet_replica.cc:330] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba: stopping tablet replica
I20260923 17:34:45.352973 23471 tablet_service.cc:1514] Processing DeleteTablet for tablet 5767ae77f6214c038014baf583554e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:45 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:40750
I20260923 17:34:45.353191 23792 raft_consensus.cc:2229] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:34:45.353274 23790 tablet_replica.cc:330] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935: stopping tablet replica
I20260923 17:34:45.353492 23792 raft_consensus.cc:2258] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:45.353484 23790 raft_consensus.cc:2229] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:34:45.354081 23790 raft_consensus.cc:2258] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:45.353962 23193 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:34:45.354669 23193 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:34:45.354703 23792 ts_tablet_manager.cc:1905] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:45.355039 23332 tablet_service.cc:1514] Processing DeleteTablet for tablet 5767ae77f6214c038014baf583554e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:45 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:50358
I20260923 17:34:45.355369 23791 tablet_replica.cc:330] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294: stopping tablet replica
I20260923 17:34:45.355567 23791 raft_consensus.cc:2229] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:34:45.355675 23790 ts_tablet_manager.cc:1905] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:45.355753 23791 raft_consensus.cc:2258] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:45.360102 23791 ts_tablet_manager.cc:1905] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:45.361099 23792 ts_tablet_manager.cc:1918] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:34:45.361253 23792 log.cc:1196] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/5767ae77f6214c038014baf583554e40
I20260923 17:34:45.361672 23792 ts_tablet_manager.cc:1939] T 5767ae77f6214c038014baf583554e40 P 757ca798cea6407eb879293c28b026ba: Deleting consensus metadata
I20260923 17:34:45.362897 23182 catalog_manager.cc:4841] TS 757ca798cea6407eb879293c28b026ba (127.20.240.67:38305): tablet 5767ae77f6214c038014baf583554e40 (table test-table [id=c327a583ec694aa4b420a6ba9708f066]) successfully deleted
I20260923 17:34:45.365048 23790 ts_tablet_manager.cc:1918] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:34:45.365293 23790 log.cc:1196] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/5767ae77f6214c038014baf583554e40
I20260923 17:34:45.365716 23790 ts_tablet_manager.cc:1939] T 5767ae77f6214c038014baf583554e40 P ca70020b320847939189b92deabb4935: Deleting consensus metadata
I20260923 17:34:45.367129 23791 ts_tablet_manager.cc:1918] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:34:45.367496 23791 log.cc:1196] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/5767ae77f6214c038014baf583554e40
I20260923 17:34:45.367933 23791 ts_tablet_manager.cc:1939] T 5767ae77f6214c038014baf583554e40 P cc54d0b0eee54fa49d96f5e63e778294: Deleting consensus metadata
I20260923 17:34:45.369006 23180 catalog_manager.cc:4841] TS ca70020b320847939189b92deabb4935 (127.20.240.66:34489): tablet 5767ae77f6214c038014baf583554e40 (table test-table [id=c327a583ec694aa4b420a6ba9708f066]) successfully deleted
I20260923 17:34:45.369223 23179 catalog_manager.cc:4841] TS cc54d0b0eee54fa49d96f5e63e778294 (127.20.240.65:38891): tablet 5767ae77f6214c038014baf583554e40 (table test-table [id=c327a583ec694aa4b420a6ba9708f066]) successfully deleted
I20260923 17:34:45.372320 23332 tablet_service.cc:1467] Processing CreateTablet for tablet bef1e4158c4d498b875f0b9e931042ed (DEFAULT_TABLE table=test-table [id=3354aa19d59e4fcd96b61ead839c80bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:45.372483 23471 tablet_service.cc:1467] Processing CreateTablet for tablet bef1e4158c4d498b875f0b9e931042ed (DEFAULT_TABLE table=test-table [id=3354aa19d59e4fcd96b61ead839c80bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:45.372641 23610 tablet_service.cc:1467] Processing CreateTablet for tablet bef1e4158c4d498b875f0b9e931042ed (DEFAULT_TABLE table=test-table [id=3354aa19d59e4fcd96b61ead839c80bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:34:45.372833 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bef1e4158c4d498b875f0b9e931042ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:45.372969 23471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bef1e4158c4d498b875f0b9e931042ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:45.373018 23610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bef1e4158c4d498b875f0b9e931042ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:45.379628 23803 tablet_bootstrap.cc:492] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba: Bootstrap starting.
I20260923 17:34:45.381320 23803 tablet_bootstrap.cc:654] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:45.381595 23801 tablet_bootstrap.cc:492] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294: Bootstrap starting.
I20260923 17:34:45.383219 23801 tablet_bootstrap.cc:654] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:45.383697 23803 tablet_bootstrap.cc:492] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba: No bootstrap required, opened a new log
I20260923 17:34:45.383927 23803 ts_tablet_manager.cc:1397] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260923 17:34:45.384073 23802 tablet_bootstrap.cc:492] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935: Bootstrap starting.
I20260923 17:34:45.384755 23803 raft_consensus.cc:348] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.385022 23803 raft_consensus.cc:374] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:45.385097 23803 raft_consensus.cc:729] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757ca798cea6407eb879293c28b026ba, State: Initialized, Role: FOLLOWER
I20260923 17:34:45.385390 23803 consensus_queue.cc:260] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.385872 23802 tablet_bootstrap.cc:654] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:45.388863 23803 ts_tablet_manager.cc:1428] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260923 17:34:45.394958 23801 tablet_bootstrap.cc:492] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294: No bootstrap required, opened a new log
I20260923 17:34:45.395215 23801 ts_tablet_manager.cc:1397] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260923 17:34:45.396066 23801 raft_consensus.cc:348] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.396268 23802 tablet_bootstrap.cc:492] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935: No bootstrap required, opened a new log
I20260923 17:34:45.396410 23801 raft_consensus.cc:374] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:45.396533 23801 raft_consensus.cc:729] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc54d0b0eee54fa49d96f5e63e778294, State: Initialized, Role: FOLLOWER
I20260923 17:34:45.396545 23802 ts_tablet_manager.cc:1397] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260923 17:34:45.396859 23801 consensus_queue.cc:260] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.397564 23801 ts_tablet_manager.cc:1428] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 17:34:45.397497 23802 raft_consensus.cc:348] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.398047 23802 raft_consensus.cc:374] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:45.398175 23802 raft_consensus.cc:729] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca70020b320847939189b92deabb4935, State: Initialized, Role: FOLLOWER
I20260923 17:34:45.398442 23802 consensus_queue.cc:260] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.399040 23802 ts_tablet_manager.cc:1428] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260923 17:34:45.498039 23777 raft_consensus.cc:482] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:34:45.498332 23777 raft_consensus.cc:504] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.499058 23777 leader_election.cc:288] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc54d0b0eee54fa49d96f5e63e778294 (127.20.240.65:38891), 757ca798cea6407eb879293c28b026ba (127.20.240.67:38305)
I20260923 17:34:45.499711 23351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef1e4158c4d498b875f0b9e931042ed" candidate_uuid: "ca70020b320847939189b92deabb4935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc54d0b0eee54fa49d96f5e63e778294" is_pre_election: true
I20260923 17:34:45.499711 23629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef1e4158c4d498b875f0b9e931042ed" candidate_uuid: "ca70020b320847939189b92deabb4935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757ca798cea6407eb879293c28b026ba" is_pre_election: true
I20260923 17:34:45.499997 23629 raft_consensus.cc:2454] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca70020b320847939189b92deabb4935 in term 0.
I20260923 17:34:45.499997 23351 raft_consensus.cc:2454] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca70020b320847939189b92deabb4935 in term 0.
I20260923 17:34:45.500463 23422 leader_election.cc:302] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca70020b320847939189b92deabb4935, cc54d0b0eee54fa49d96f5e63e778294; no voters:
I20260923 17:34:45.500813 23777 raft_consensus.cc:2781] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:34:45.500941 23777 raft_consensus.cc:482] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:34:45.501058 23777 raft_consensus.cc:3037] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:45.504428 23777 raft_consensus.cc:504] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.505218 23777 leader_election.cc:288] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [CANDIDATE]: Term 1 election: Requested vote from peers cc54d0b0eee54fa49d96f5e63e778294 (127.20.240.65:38891), 757ca798cea6407eb879293c28b026ba (127.20.240.67:38305)
I20260923 17:34:45.505864 23629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef1e4158c4d498b875f0b9e931042ed" candidate_uuid: "ca70020b320847939189b92deabb4935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757ca798cea6407eb879293c28b026ba"
I20260923 17:34:45.505853 23351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef1e4158c4d498b875f0b9e931042ed" candidate_uuid: "ca70020b320847939189b92deabb4935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc54d0b0eee54fa49d96f5e63e778294"
I20260923 17:34:45.506085 23351 raft_consensus.cc:3037] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:45.506089 23629 raft_consensus.cc:3037] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:45.508378 23629 raft_consensus.cc:2454] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca70020b320847939189b92deabb4935 in term 1.
I20260923 17:34:45.508891 23425 leader_election.cc:302] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757ca798cea6407eb879293c28b026ba, ca70020b320847939189b92deabb4935; no voters:
I20260923 17:34:45.509171 23777 raft_consensus.cc:2781] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:45.509276 23351 raft_consensus.cc:2454] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca70020b320847939189b92deabb4935 in term 1.
I20260923 17:34:45.509433 23777 raft_consensus.cc:686] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [term 1 LEADER]: Becoming Leader. State: Replica: ca70020b320847939189b92deabb4935, State: Running, Role: LEADER
I20260923 17:34:45.509845 23777 consensus_queue.cc:237] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } }
I20260923 17:34:45.512727 23195 catalog_manager.cc:5495] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca70020b320847939189b92deabb4935 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "ca70020b320847939189b92deabb4935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca70020b320847939189b92deabb4935" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 34489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 } health_report { overall_health: UNKNOWN } } }
I20260923 17:34:45.570549 23351 raft_consensus.cc:1261] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Refusing update from remote peer ca70020b320847939189b92deabb4935: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:34:45.570641 23629 raft_consensus.cc:1261] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Refusing update from remote peer ca70020b320847939189b92deabb4935: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:34:45.571095 23777 consensus_queue.cc:1035] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc54d0b0eee54fa49d96f5e63e778294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 38891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:34:45.571384 23723 consensus_queue.cc:1035] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "757ca798cea6407eb879293c28b026ba" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:34:45.594556 23769 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 17:34:47.194100 23194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:34:47.194319 23194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:34:47.200147 23194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e9ab21fea16458eb413929177ed7820: Sending DeleteTablet for 3 replicas of tablet bef1e4158c4d498b875f0b9e931042ed
I20260923 17:34:47.201066 23471 tablet_service.cc:1514] Processing DeleteTablet for tablet bef1e4158c4d498b875f0b9e931042ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:47 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:40750
I20260923 17:34:47.201273 23610 tablet_service.cc:1514] Processing DeleteTablet for tablet bef1e4158c4d498b875f0b9e931042ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:47 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:35552
I20260923 17:34:47.201297 23332 tablet_service.cc:1514] Processing DeleteTablet for tablet bef1e4158c4d498b875f0b9e931042ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:34:47 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:50358
I20260923 17:34:47.201999 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 23266
I20260923 17:34:47.235867 23845 tablet_replica.cc:330] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935: stopping tablet replica
I20260923 17:34:47.236119 23845 raft_consensus.cc:2229] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:34:47.236542 23845 raft_consensus.cc:2258] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:47.238596 23845 ts_tablet_manager.cc:1905] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:47.242195 23846 tablet_replica.cc:330] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba: stopping tablet replica
I20260923 17:34:47.242304 23847 tablet_replica.cc:330] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294: stopping tablet replica
I20260923 17:34:47.242448 23846 raft_consensus.cc:2229] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:34:47.242511 23847 raft_consensus.cc:2229] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:34:47.242792 23846 raft_consensus.cc:2258] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:47.243003 23847 raft_consensus.cc:2258] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:34:47.244062 23847 ts_tablet_manager.cc:1905] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:34:47.244257 23846 ts_tablet_manager.cc:1905] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 17:34:47 dist-test-slave-ppjl krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:34:47.267367 23846 ts_tablet_manager.cc:1918] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:34:47.267670 23849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:47.203710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50382 (local address 127.20.240.65:38891)
0923 17:34:47.228417 (+ 24707us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:47.228424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:47.229369 (+ 945us) server_negotiation.cc:407] Connection header received
0923 17:34:47.229448 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:47.229454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:47.229538 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:47.229639 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:47.231597 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:47.233049 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:47.237537 (+ 4488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:47.238310 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:47.238489 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:47.255268 (+ 16779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:47.255288 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:47.255306 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:47.255357 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:47.264026 (+ 8669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:47.264941 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:47.264949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:47.264952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:47.265223 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:47.265740 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:47.265744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:47.265746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:47.266251 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:47.266673 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:47.266845 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 17:34:47.267095 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24405,"thread_start_us":30345,"threads_started":1}
I20260923 17:34:47.268112 23846 log.cc:1196] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/bef1e4158c4d498b875f0b9e931042ed
I20260923 17:34:47.268666 23847 ts_tablet_manager.cc:1918] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:34:47.268919 23847 log.cc:1196] T bef1e4158c4d498b875f0b9e931042ed P cc54d0b0eee54fa49d96f5e63e778294: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/bef1e4158c4d498b875f0b9e931042ed
I20260923 17:34:47.269066 23845 ts_tablet_manager.cc:1918] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:34:47.269312 23845 log.cc:1196] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/bef1e4158c4d498b875f0b9e931042ed
W20260923 17:34:47.268965 23397 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50382
I20260923 17:34:47.269912 23846 ts_tablet_manager.cc:1939] T bef1e4158c4d498b875f0b9e931042ed P 757ca798cea6407eb879293c28b026ba: Deleting consensus metadata
I20260923 17:34:47.270529 23845 ts_tablet_manager.cc:1939] T bef1e4158c4d498b875f0b9e931042ed P ca70020b320847939189b92deabb4935: Deleting consensus metadata
W20260923 17:34:47.270695 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 23266: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:34:47.271116 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 23266
I20260923 17:34:47.271905 23182 catalog_manager.cc:4841] TS 757ca798cea6407eb879293c28b026ba (127.20.240.67:38305): tablet bef1e4158c4d498b875f0b9e931042ed (table test-table [id=3354aa19d59e4fcd96b61ead839c80bd]) successfully deleted
I20260923 17:34:47.278167 23180 catalog_manager.cc:4841] TS ca70020b320847939189b92deabb4935 (127.20.240.66:34489): tablet bef1e4158c4d498b875f0b9e931042ed (table test-table [id=3354aa19d59e4fcd96b61ead839c80bd]) successfully deleted
W20260923 17:34:47.292192 23179 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.240.65:38891 (error 108)
I20260923 17:34:47.292429 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 23406
I20260923 17:34:47.294553 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:47.293700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:38891 (local address 127.0.0.1:50384)
0923 17:34:47.294169 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:47.294337 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":245,"threads_started":1}
W20260923 17:34:47.294824 23179 catalog_manager.cc:4584] TS cc54d0b0eee54fa49d96f5e63e778294 (127.20.240.65:38891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bef1e4158c4d498b875f0b9e931042ed: Network error: Client connection negotiation failed: client connection to 127.20.240.65:38891: connect: Connection refused (error 111)
Sep 23 17:34:47 dist-test-slave-ppjl krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:34:47.308173 23851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:47.293328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.20.240.66:34489)
0923 17:34:47.294068 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:47.294074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:47.294092 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:34:47.294160 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:47.294164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:47.294232 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:47.294335 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:47.295959 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:47.296688 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:47.298005 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:47.298424 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:47.298531 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:47.302552 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:47.302567 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:47.302588 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:47.302628 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:47.305441 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:47.306110 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:47.306118 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:47.306120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:47.306251 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:47.306601 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:47.306605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:47.306607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:47.307001 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:47.307304 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:47.307621 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 17:34:47.307845 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":531,"threads_started":1}
W20260923 17:34:47.308810 23536 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40790
W20260923 17:34:47.309480 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 23406: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:34:47.309617 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 23406
I20260923 17:34:47.332162 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 23545
Sep 23 17:34:47 dist-test-slave-ppjl krb5kdc[23150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:34:47.348474 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:47.347981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:38891 (local address 127.0.0.1:50400)
0923 17:34:47.348194 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:47.348283 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 17:34:47.349529 23852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:47.349146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:38891 (local address 127.0.0.1:50416)
0923 17:34:47.349296 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:47.349386 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 17:34:47.349524 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:47.333018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.20.240.67:38305)
0923 17:34:47.333540 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:47.333545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:47.333562 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:34:47.333620 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:47.333624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:47.333690 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:47.333784 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:47.335091 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:47.335919 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:47.337202 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:47.337714 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:47.337835 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:47.342930 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:47.342946 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:47.342963 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:47.343048 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:47.346339 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:47.347111 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:47.347121 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:47.347123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:47.347377 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:47.347811 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:47.347817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:47.347820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:47.348247 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:47.348541 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:47.348894 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 17:34:47.349139 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":259,"threads_started":1}
W20260923 17:34:47.350236 23675 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35562
W20260923 17:34:47.350869 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 23545: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:34:47.351024 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 23545
I20260923 17:34:47.368489 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 23163
I20260923 17:34:47.369717 23235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:34:47.494987 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 23163
2026-09-23T17:34:47Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8527 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:34:47 dist-test-slave-ppjl krb5kdc[23857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:34:47 dist-test-slave-ppjl krb5kdc[23857](info): set up 2 sockets
Sep 23 17:34:47 dist-test-slave-ppjl krb5kdc[23857](info): commencing operation
krb5kdc: starting...
W20260923 17:34:48.908041 21441 mini_kdc.cc:121] Time spent starting KDC: real 1.372s user 0.006s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:34:49 dist-test-slave-ppjl krb5kdc[23857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:34:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:34:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:34:49.103507 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:40919
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:40919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:49.431398 23870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:49.431707 23870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:49.431804 23870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:49.439532 23870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:34:49.439664 23870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:49.439720 23870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:49.439776 23870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:34:49.439819 23870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:34:49.450034 23870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:40919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:40919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:49.451396 23870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:49.452683 23870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:49.464856 23876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:49.464776 23875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:49.465291 23878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:49.465684 23870 server_base.cc:1034] running on GCE node
I20260923 17:34:49.466516 23870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:49.468092 23870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:49.469298 23870 hybrid_clock.cc:648] HybridClock initialized: now 1790184889469248 us; error 42 us; skew 500 ppm
I20260923 17:34:49.469651 23870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:49 dist-test-slave-ppjl krb5kdc[23857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184889, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:49.472873 23870 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:34:49.474422 23870 webserver.cc:460] Webserver started at http://127.20.240.126:38747/ using document root <none> and password file <none>
I20260923 17:34:49.475124 23870 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:49.475268 23870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:49.475550 23870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:49.478480 23870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "cbf2a9366e554f8a9a8ef652ae99c1d3"
format_stamp: "Formatted at 2026-09-23 17:34:49 on dist-test-slave-ppjl"
server_key: "ce8f8c28ecea20b5c5945411bbfc2109"
server_key_iv: "f956b4cfff92dc9450a96d993f598ccd"
server_key_version: "encryptionkey@0"
I20260923 17:34:49.479213 23870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "cbf2a9366e554f8a9a8ef652ae99c1d3"
format_stamp: "Formatted at 2026-09-23 17:34:49 on dist-test-slave-ppjl"
server_key: "ce8f8c28ecea20b5c5945411bbfc2109"
server_key_iv: "f956b4cfff92dc9450a96d993f598ccd"
server_key_version: "encryptionkey@0"
I20260923 17:34:49.484483 23870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:34:49.487870 23885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:49.489219 23870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:34:49.489392 23870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "cbf2a9366e554f8a9a8ef652ae99c1d3"
format_stamp: "Formatted at 2026-09-23 17:34:49 on dist-test-slave-ppjl"
server_key: "ce8f8c28ecea20b5c5945411bbfc2109"
server_key_iv: "f956b4cfff92dc9450a96d993f598ccd"
server_key_version: "encryptionkey@0"
I20260923 17:34:49.489578 23870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:49.512946 23870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:49.517359 23870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:49.517647 23870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:49.535841 23870 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:40919
I20260923 17:34:49.535841 23936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:40919 every 8 connection(s)
I20260923 17:34:49.537269 23870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:34:49.540932 23937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:49.541803 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 23870
I20260923 17:34:49.542009 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:34:49.542575 21441 external_mini_cluster.cc:1428] Setting key e4a5a602c6c00a9fefbe7e3b91d60b23
I20260923 17:34:49.554913 23937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3: Bootstrap starting.
Sep 23 17:34:49 dist-test-slave-ppjl krb5kdc[23857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:49.559073 23937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:49.560453 23937 log.cc:824] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:49.564364 23937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3: No bootstrap required, opened a new log
I20260923 17:34:49.566675 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:49.544498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48054 (local address 127.20.240.126:40919)
0923 17:34:49.545282 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:49.545294 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:49.545361 (+ 67us) server_negotiation.cc:407] Connection header received
0923 17:34:49.546908 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:49.546933 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:49.547381 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:49.547704 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:49.549451 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:49.550756 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:49.552590 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:49.553295 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:49.553439 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:49.558591 (+ 5152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:49.558600 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:49.558616 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:49.558662 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:49.562177 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:49.563205 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:49.563215 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:49.563225 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:49.563421 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:49.563920 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:49.563923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:49.563925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:49.564446 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:49.564723 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:49.565183 (+ 460us) server_negotiation.cc:299] Negotiation successful
0923 17:34:49.565614 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":359,"threads_started":1}
I20260923 17:34:49.572745 23937 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf2a9366e554f8a9a8ef652ae99c1d3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40919 } }
I20260923 17:34:49.573186 23937 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:49.573302 23937 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf2a9366e554f8a9a8ef652ae99c1d3, State: Initialized, Role: FOLLOWER
I20260923 17:34:49.573902 23937 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf2a9366e554f8a9a8ef652ae99c1d3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40919 } }
I20260923 17:34:49.574239 23937 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:34:49.574399 23937 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:34:49.574630 23937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:49.577181 23937 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf2a9366e554f8a9a8ef652ae99c1d3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40919 } }
I20260923 17:34:49.577797 23937 leader_election.cc:302] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbf2a9366e554f8a9a8ef652ae99c1d3; no voters:
I20260923 17:34:49.578522 23937 leader_election.cc:288] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:34:49.578660 23942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:49.579701 23942 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [term 1 LEADER]: Becoming Leader. State: Replica: cbf2a9366e554f8a9a8ef652ae99c1d3, State: Running, Role: LEADER
I20260923 17:34:49.580292 23942 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf2a9366e554f8a9a8ef652ae99c1d3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40919 } }
I20260923 17:34:49.580886 23937 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:34:49.583125 23943 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbf2a9366e554f8a9a8ef652ae99c1d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf2a9366e554f8a9a8ef652ae99c1d3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40919 } } }
I20260923 17:34:49.583087 23944 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbf2a9366e554f8a9a8ef652ae99c1d3. Latest consensus state: current_term: 1 leader_uuid: "cbf2a9366e554f8a9a8ef652ae99c1d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf2a9366e554f8a9a8ef652ae99c1d3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40919 } } }
I20260923 17:34:49.583544 23943 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:49.583544 23944 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:49.584446 23949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:34:49.589960 23949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:34:49.600272 23949 catalog_manager.cc:1347] Generated new cluster ID: 5b8949d4cda2495c914051db1824b311
I20260923 17:34:49.600414 23949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:34:49.610747 23949 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:34:49.612371 23949 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:34:49.624421 23949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbf2a9366e554f8a9a8ef652ae99c1d3: Generated new TSK 0
I20260923 17:34:49.625393 23949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:34:49.678694 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:40919
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:50.013021 23963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:50.013336 23963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:50.013466 23963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:50.020946 23963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:50.021093 23963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:50.021255 23963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:34:50.031869 23963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:40919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:50.033319 23963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:50.034718 23963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:50.048836 23968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:50.049511 23969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:50.050208 23971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:50.050997 23963 server_base.cc:1034] running on GCE node
I20260923 17:34:50.051764 23963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:50.052814 23963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:50.054065 23963 hybrid_clock.cc:648] HybridClock initialized: now 1790184890054008 us; error 46 us; skew 500 ppm
I20260923 17:34:50.054441 23963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:50 dist-test-slave-ppjl krb5kdc[23857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184890, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:50.058188 23963 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:34:50.059991 23963 webserver.cc:460] Webserver started at http://127.20.240.65:37033/ using document root <none> and password file <none>
I20260923 17:34:50.060765 23963 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:50.060909 23963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:50.061221 23963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:50.064409 23963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "eccda08dafee4a2787c2015730da89d8"
format_stamp: "Formatted at 2026-09-23 17:34:50 on dist-test-slave-ppjl"
server_key: "ed87f06ccd96d581886e0373c63c8d33"
server_key_iv: "efe96a8e28029915b4d761630a73a6a9"
server_key_version: "encryptionkey@0"
I20260923 17:34:50.065256 23963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "eccda08dafee4a2787c2015730da89d8"
format_stamp: "Formatted at 2026-09-23 17:34:50 on dist-test-slave-ppjl"
server_key: "ed87f06ccd96d581886e0373c63c8d33"
server_key_iv: "efe96a8e28029915b4d761630a73a6a9"
server_key_version: "encryptionkey@0"
I20260923 17:34:50.071295 23963 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260923 17:34:50.075449 23978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:50.077023 23963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:34:50.077216 23963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "eccda08dafee4a2787c2015730da89d8"
format_stamp: "Formatted at 2026-09-23 17:34:50 on dist-test-slave-ppjl"
server_key: "ed87f06ccd96d581886e0373c63c8d33"
server_key_iv: "efe96a8e28029915b4d761630a73a6a9"
server_key_version: "encryptionkey@0"
I20260923 17:34:50.077425 23963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:50.105685 23963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:50.110263 23963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:50.110540 23963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:50.111564 23963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:34:50.113077 23963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:50.113161 23963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:50.113286 23963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:50.113341 23963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:50.147261 23963 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:36625
I20260923 17:34:50.147317 24090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:36625 every 8 connection(s)
I20260923 17:34:50.148528 23963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:34:50.150020 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 23963
I20260923 17:34:50.150204 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:34:50.150990 21441 external_mini_cluster.cc:1428] Setting key c7adda46e7bcffaba2442959ec16a719
Sep 23 17:34:50 dist-test-slave-ppjl krb5kdc[23857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184890, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
I20260923 17:34:50.170233 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:50.151043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:54377 (local address 127.20.240.126:40919)
0923 17:34:50.151947 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:50.151954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:50.152795 (+ 841us) server_negotiation.cc:407] Connection header received
0923 17:34:50.154793 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:50.154799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:50.154885 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:50.154982 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:50.156656 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:50.157650 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:50.159411 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:50.160016 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:50.160150 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:50.164376 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:50.164382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:50.164385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:50.164434 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:50.167057 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:50.167871 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:50.167876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:50.167878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:50.168002 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:50.168447 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:50.168450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:50.168453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:50.168731 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:50.168868 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:50.169633 (+ 765us) server_negotiation.cc:299] Negotiation successful
0923 17:34:50.169869 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":653,"threads_started":1}
I20260923 17:34:50.171350 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:50.151770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:40919 (local address 127.20.240.65:54377)
0923 17:34:50.152632 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:50.152682 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:50.154508 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:50.155288 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:50.155297 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:50.155872 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:50.156469 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:50.156489 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:50.157870 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:50.157874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:50.159250 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:50.159271 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:50.160278 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:50.160286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:50.160509 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:50.161415 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:50.161441 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:50.164055 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:50.167264 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:50.167275 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:50.167294 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:50.167729 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:50.168117 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:50.168122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:50.168126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:50.168328 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:50.168864 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:50.168873 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:50.169237 (+ 364us) client_negotiation.cc:769] Sending connection context
0923 17:34:50.169496 (+ 259us) client_negotiation.cc:240] Negotiation successful
0923 17:34:50.170095 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":319,"threads_started":1}
I20260923 17:34:50.173202 24091 heartbeater.cc:347] Connected to a master server at 127.20.240.126:40919
I20260923 17:34:50.173545 24091 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:50.174342 24091 heartbeater.cc:511] Master 127.20.240.126:40919 requested a full tablet report, sending...
I20260923 17:34:50.176553 23902 ts_manager.cc:194] Registered new tserver with Master: eccda08dafee4a2787c2015730da89d8 (127.20.240.65:36625)
I20260923 17:34:50.178059 23902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:54377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:34:50.196264 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:40919
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:50.520834 24097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:50.521136 24097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:50.521236 24097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:50.528815 24097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:50.528944 24097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:50.529059 24097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:34:50.541386 24097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:40919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:50.542963 24097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:50.544664 24097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:50.560575 24103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:50.560504 24105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:50.560549 24102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:50.561596 24097 server_base.cc:1034] running on GCE node
I20260923 17:34:50.562526 24097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:50.563731 24097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:50.564973 24097 hybrid_clock.cc:648] HybridClock initialized: now 1790184890564924 us; error 47 us; skew 500 ppm
I20260923 17:34:50.565313 24097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:50 dist-test-slave-ppjl krb5kdc[23857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184890, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:50.568819 24097 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:34:50.570681 24097 webserver.cc:460] Webserver started at http://127.20.240.66:45573/ using document root <none> and password file <none>
I20260923 17:34:50.571556 24097 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:50.571703 24097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:50.572055 24097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:50.575603 24097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "bbc3e5de5a6e4ab8a6d5b12fe3c98e92"
format_stamp: "Formatted at 2026-09-23 17:34:50 on dist-test-slave-ppjl"
server_key: "374d28b816058a0bd749fcb57a674961"
server_key_iv: "638312f76ec1dd6ff1e757af5303c2ed"
server_key_version: "encryptionkey@0"
I20260923 17:34:50.576596 24097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "bbc3e5de5a6e4ab8a6d5b12fe3c98e92"
format_stamp: "Formatted at 2026-09-23 17:34:50 on dist-test-slave-ppjl"
server_key: "374d28b816058a0bd749fcb57a674961"
server_key_iv: "638312f76ec1dd6ff1e757af5303c2ed"
server_key_version: "encryptionkey@0"
I20260923 17:34:50.582921 24097 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260923 17:34:50.587107 24112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:50.588797 24097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:34:50.588973 24097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "bbc3e5de5a6e4ab8a6d5b12fe3c98e92"
format_stamp: "Formatted at 2026-09-23 17:34:50 on dist-test-slave-ppjl"
server_key: "374d28b816058a0bd749fcb57a674961"
server_key_iv: "638312f76ec1dd6ff1e757af5303c2ed"
server_key_version: "encryptionkey@0"
I20260923 17:34:50.589191 24097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:50.612586 24097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:50.617100 24097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:50.617384 24097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:50.618407 24097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:34:50.619844 24097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:50.619922 24097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:50.620025 24097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:50.620076 24097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:50.651292 24097 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:41709
I20260923 17:34:50.651467 24224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:41709 every 8 connection(s)
I20260923 17:34:50.652602 24097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:34:50.654920 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 24097
I20260923 17:34:50.655089 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:34:50.655610 21441 external_mini_cluster.cc:1428] Setting key 1d6702923c2fa021fd63d69f504d634b
Sep 23 17:34:50 dist-test-slave-ppjl krb5kdc[23857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184890, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
I20260923 17:34:50.678009 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:50.655566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:48517 (local address 127.20.240.126:40919)
0923 17:34:50.655786 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:50.655793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:50.657347 (+ 1554us) server_negotiation.cc:407] Connection header received
0923 17:34:50.659646 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:50.659653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:50.659751 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:50.659869 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:50.661893 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:50.662911 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:50.665478 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:50.665921 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:50.666030 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:50.670156 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:50.670164 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:50.670169 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:50.670218 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:50.673693 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:50.674891 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:50.674896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:50.674898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:50.675057 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:50.675856 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:50.675861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:50.675863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:50.676211 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:50.676344 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:50.677424 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0923 17:34:50.677659 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:34:50.679112 24227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:50.656204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:40919 (local address 127.20.240.66:48517)
0923 17:34:50.657156 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:50.657213 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:50.659106 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:50.660209 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:50.660220 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:50.660889 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:50.661677 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:50.661705 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:50.663088 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:50.663094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:50.665226 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:50.665243 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:50.666151 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:50.666157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:50.666445 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:50.667229 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:50.667260 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:50.669810 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:50.673922 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:50.673931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:50.673948 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:50.674615 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:50.675315 (+ 700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:50.675320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:50.675324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:50.675680 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:50.676485 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:50.676492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:50.676974 (+ 482us) client_negotiation.cc:769] Sending connection context
0923 17:34:50.677248 (+ 274us) client_negotiation.cc:240] Negotiation successful
0923 17:34:50.677879 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":374,"threads_started":1}
I20260923 17:34:50.680886 24225 heartbeater.cc:347] Connected to a master server at 127.20.240.126:40919
I20260923 17:34:50.681268 24225 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:50.682241 24225 heartbeater.cc:511] Master 127.20.240.126:40919 requested a full tablet report, sending...
I20260923 17:34:50.684186 23902 ts_manager.cc:194] Registered new tserver with Master: bbc3e5de5a6e4ab8a6d5b12fe3c98e92 (127.20.240.66:41709)
I20260923 17:34:50.685137 23902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:48517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:34:50.703143 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:40919
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:39375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:51.027632 24230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:51.027997 24230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:51.028134 24230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:51.036290 24230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:51.036439 24230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:51.036589 24230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:34:51.046689 24230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:39375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:40919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:51.048107 24230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:51.049500 24230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:51.063422 24235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:51.063336 24236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:51.063372 24238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:51.064074 24230 server_base.cc:1034] running on GCE node
I20260923 17:34:51.064709 24230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:51.065712 24230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:51.066941 24230 hybrid_clock.cc:648] HybridClock initialized: now 1790184891066881 us; error 52 us; skew 500 ppm
I20260923 17:34:51.067305 24230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:51 dist-test-slave-ppjl krb5kdc[23857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184891, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:51.070541 24230 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:34:51.072154 24230 webserver.cc:460] Webserver started at http://127.20.240.67:44761/ using document root <none> and password file <none>
I20260923 17:34:51.072858 24230 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:51.072969 24230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:51.073246 24230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:51.076375 24230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "adcab9ccd3b14d8888eb7334100fae72"
format_stamp: "Formatted at 2026-09-23 17:34:51 on dist-test-slave-ppjl"
server_key: "525ac75986e712fb75f427a240e61af0"
server_key_iv: "7411862e0837c7496bf2524302b273c4"
server_key_version: "encryptionkey@0"
I20260923 17:34:51.077136 24230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "adcab9ccd3b14d8888eb7334100fae72"
format_stamp: "Formatted at 2026-09-23 17:34:51 on dist-test-slave-ppjl"
server_key: "525ac75986e712fb75f427a240e61af0"
server_key_iv: "7411862e0837c7496bf2524302b273c4"
server_key_version: "encryptionkey@0"
I20260923 17:34:51.082480 24230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:34:51.086392 24245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:51.088008 24230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:34:51.088163 24230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "adcab9ccd3b14d8888eb7334100fae72"
format_stamp: "Formatted at 2026-09-23 17:34:51 on dist-test-slave-ppjl"
server_key: "525ac75986e712fb75f427a240e61af0"
server_key_iv: "7411862e0837c7496bf2524302b273c4"
server_key_version: "encryptionkey@0"
I20260923 17:34:51.088366 24230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:51.102748 24230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:51.107012 24230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:51.107316 24230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:51.108242 24230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:34:51.109537 24230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:51.109611 24230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:51.109700 24230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:51.109745 24230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:51.140990 24230 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:43525
I20260923 17:34:51.141002 24357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:43525 every 8 connection(s)
I20260923 17:34:51.142369 24230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:34:51.151518 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 24230
I20260923 17:34:51.151721 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:34:51.152243 21441 external_mini_cluster.cc:1428] Setting key 7870ed73accd38d15fde0d886acc30da
Sep 23 17:34:51 dist-test-slave-ppjl krb5kdc[23857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184891, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:51.163322 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:51.145010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:50995 (local address 127.20.240.126:40919)
0923 17:34:51.145210 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:51.145216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:51.146406 (+ 1190us) server_negotiation.cc:407] Connection header received
0923 17:34:51.148377 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:51.148383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:51.148479 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:51.148586 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:51.150141 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:51.151299 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:51.153387 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:51.153944 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:51.154045 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:51.157653 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:51.157658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:51.157661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:51.157705 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:51.160357 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:51.161038 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:51.161042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:51.161044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:51.161183 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:51.161658 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:51.161662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:51.161664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:51.161987 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:51.162109 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:51.162852 (+ 743us) server_negotiation.cc:299] Negotiation successful
0923 17:34:51.163030 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:34:51.164268 24360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:51.145493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:40919 (local address 127.20.240.67:50995)
0923 17:34:51.146230 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:51.146269 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:51.148068 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:51.148835 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:51.148845 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:51.149398 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:51.149958 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:51.149975 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:51.151483 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:51.151489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:51.153224 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:51.153238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:51.154200 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:51.154210 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:51.154500 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:51.155241 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:51.155273 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:51.157442 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:51.160489 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:51.160496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:51.160510 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:51.160927 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:51.161311 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:51.161315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:51.161317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:51.161514 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:51.162103 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:51.162110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:51.162496 (+ 386us) client_negotiation.cc:769] Sending connection context
0923 17:34:51.162735 (+ 239us) client_negotiation.cc:240] Negotiation successful
0923 17:34:51.163142 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":277,"threads_started":1}
I20260923 17:34:51.165813 24358 heartbeater.cc:347] Connected to a master server at 127.20.240.126:40919
I20260923 17:34:51.166157 24358 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:51.167001 24358 heartbeater.cc:511] Master 127.20.240.126:40919 requested a full tablet report, sending...
I20260923 17:34:51.168520 23902 ts_manager.cc:194] Registered new tserver with Master: adcab9ccd3b14d8888eb7334100fae72 (127.20.240.67:43525)
I20260923 17:34:51.169305 23902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:50995
I20260923 17:34:51.181174 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:34:51.181445 24091 heartbeater.cc:503] Master 127.20.240.126:40919 was elected leader, sending a full tablet report...
W20260923 17:34:51.211236 24093 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:34:51.208630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48058 (local address 127.20.240.126:40919)
0923 17:34:51.208859 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:51.208865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:51.209244 (+ 379us) server_negotiation.cc:407] Connection header received
0923 17:34:51.209433 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:51.209437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:51.209503 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:51.209587 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:51.210953 (+ 1366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48058: BlockingRecv error: recv got EOF from 127.0.0.1:48058 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:34:51.214148 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 23963
W20260923 17:34:51.217015 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 23963: Not authorized: Client connection negotiation failed: client connection to 127.20.240.65:36625: server requires authentication, but client does not have Kerberos credentials available
I20260923 17:34:51.217227 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 23963
I20260923 17:34:51.229724 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 24097
W20260923 17:34:51.232370 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 24097: Not authorized: Client connection negotiation failed: client connection to 127.20.240.66:41709: server requires authentication, but client does not have Kerberos credentials available
I20260923 17:34:51.232542 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 24097
I20260923 17:34:51.245335 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 24230
W20260923 17:34:51.248469 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 24230: Not authorized: Client connection negotiation failed: client connection to 127.20.240.67:43525: server requires authentication, but client does not have Kerberos credentials available
I20260923 17:34:51.248620 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 24230
W20260923 17:34:51.248724 24371 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:34:51.246354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.20.240.67:43525)
0923 17:34:51.247011 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:51.247018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:51.247048 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:34:51.247110 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:51.247117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:51.247380 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:51.247539 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:51.248398 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40430: BlockingRecv error: recv got EOF from 127.0.0.1:40430 (error 108)
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":353,"threads_started":1}
I20260923 17:34:51.260656 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 23870
I20260923 17:34:51.262097 23932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:34:51.364655 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 23870
2026-09-23T17:34:51Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3865 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:34:51 dist-test-slave-ppjl krb5kdc[24375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:34:51 dist-test-slave-ppjl krb5kdc[24375](info): set up 2 sockets
Sep 23 17:34:51 dist-test-slave-ppjl krb5kdc[24375](info): commencing operation
krb5kdc: starting...
W20260923 17:34:52.368036 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:34:52 dist-test-slave-ppjl krb5kdc[24375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:34:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:34:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:34:52.560115 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:34479
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:34781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:34479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:52.882526 24388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:52.882853 24388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:52.882977 24388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:52.890306 24388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:34:52.890445 24388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:52.890523 24388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:52.890596 24388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:34:52.890661 24388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:34:52.900655 24388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:34781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:34479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:34479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:52.901999 24388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:52.903308 24388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:52.914921 24396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:52.915004 24394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:52.914980 24393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:52.915617 24388 server_base.cc:1034] running on GCE node
I20260923 17:34:52.916456 24388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:52.917964 24388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:52.919224 24388 hybrid_clock.cc:648] HybridClock initialized: now 1790184892919171 us; error 32 us; skew 500 ppm
I20260923 17:34:52.919631 24388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:52 dist-test-slave-ppjl krb5kdc[24375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184892, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:52.926733 24388 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:34:52.928342 24388 webserver.cc:460] Webserver started at http://127.20.240.126:41581/ using document root <none> and password file <none>
I20260923 17:34:52.929148 24388 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:52.929276 24388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:52.929565 24388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:52.932646 24388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "3e51be117f014fe9b64f94de7e3815ec"
format_stamp: "Formatted at 2026-09-23 17:34:52 on dist-test-slave-ppjl"
server_key: "17b1ac1859a8aaf5dbe84d2e245429f8"
server_key_iv: "2b71302c240ac8d8bf0f0250b778b7bd"
server_key_version: "encryptionkey@0"
I20260923 17:34:52.933478 24388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "3e51be117f014fe9b64f94de7e3815ec"
format_stamp: "Formatted at 2026-09-23 17:34:52 on dist-test-slave-ppjl"
server_key: "17b1ac1859a8aaf5dbe84d2e245429f8"
server_key_iv: "2b71302c240ac8d8bf0f0250b778b7bd"
server_key_version: "encryptionkey@0"
I20260923 17:34:52.938966 24388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:34:52.942631 24403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:52.944123 24388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:34:52.944303 24388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "3e51be117f014fe9b64f94de7e3815ec"
format_stamp: "Formatted at 2026-09-23 17:34:52 on dist-test-slave-ppjl"
server_key: "17b1ac1859a8aaf5dbe84d2e245429f8"
server_key_iv: "2b71302c240ac8d8bf0f0250b778b7bd"
server_key_version: "encryptionkey@0"
I20260923 17:34:52.944495 24388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:52.985334 24388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:52.989606 24388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:52.989894 24388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:53.006836 24388 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:34479
I20260923 17:34:53.006857 24454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:34479 every 8 connection(s)
I20260923 17:34:53.008314 24388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:34:53.012060 24455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:53.018122 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 24388
I20260923 17:34:53.018359 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:34:53.018954 21441 external_mini_cluster.cc:1428] Setting key 3d9b8632738280dff1c267040e7e03d2
I20260923 17:34:53.023135 24455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec: Bootstrap starting.
I20260923 17:34:53.026871 24455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:53.028014 24455 log.cc:824] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:53.031052 24455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec: No bootstrap required, opened a new log
Sep 23 17:34:53 dist-test-slave-ppjl krb5kdc[24375](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:53.040117 24455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e51be117f014fe9b64f94de7e3815ec" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 34479 } }
I20260923 17:34:53.040581 24455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:53.040705 24455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e51be117f014fe9b64f94de7e3815ec, State: Initialized, Role: FOLLOWER
I20260923 17:34:53.041404 24455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e51be117f014fe9b64f94de7e3815ec" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 34479 } }
I20260923 17:34:53.041631 24455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:34:53.041769 24455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:34:53.041952 24455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:53.042691 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:53.020937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50768 (local address 127.20.240.126:34479)
0923 17:34:53.021765 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:53.021777 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:53.021820 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:34:53.023624 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:53.023648 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:53.024044 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:53.024377 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:53.026322 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:53.027773 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:53.029341 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:53.030000 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:53.030128 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:53.034750 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:53.034759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:53.034776 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:53.034834 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:53.038021 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:53.038956 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:53.038963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:53.038970 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:53.039203 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:53.039622 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:53.039626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:53.039628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:53.040213 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:53.040455 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:53.040975 (+ 520us) server_negotiation.cc:299] Negotiation successful
0923 17:34:53.041470 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":362,"threads_started":1}
I20260923 17:34:53.044675 24455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e51be117f014fe9b64f94de7e3815ec" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 34479 } }
I20260923 17:34:53.045279 24455 leader_election.cc:302] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e51be117f014fe9b64f94de7e3815ec; no voters:
I20260923 17:34:53.045888 24455 leader_election.cc:288] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:34:53.046043 24460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:53.047106 24460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [term 1 LEADER]: Becoming Leader. State: Replica: 3e51be117f014fe9b64f94de7e3815ec, State: Running, Role: LEADER
I20260923 17:34:53.047293 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:34:53.047667 24460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e51be117f014fe9b64f94de7e3815ec" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 34479 } }
I20260923 17:34:53.048293 24455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:34:53.050396 24461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e51be117f014fe9b64f94de7e3815ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e51be117f014fe9b64f94de7e3815ec" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 34479 } } }
I20260923 17:34:53.051051 24462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e51be117f014fe9b64f94de7e3815ec. Latest consensus state: current_term: 1 leader_uuid: "3e51be117f014fe9b64f94de7e3815ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e51be117f014fe9b64f94de7e3815ec" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 34479 } } }
I20260923 17:34:53.051352 24462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:53.051785 24461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:53.062114 24466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:34:53.067705 24466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:34:53.082211 24466 catalog_manager.cc:1347] Generated new cluster ID: 8e18f5a5c4894fdba26f8029b37eda06
I20260923 17:34:53.082370 24466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:34:53.098703 24480 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:34:53.109491 24466 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:34:53.111407 24466 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:34:53.116019 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:53.104515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.20.240.126:34479)
0923 17:34:53.104722 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:53.104729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:53.107343 (+ 2614us) server_negotiation.cc:407] Connection header received
0923 17:34:53.107568 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:53.107575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:53.107666 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:53.107788 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:53.109112 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:53.110188 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:53.112127 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:53.112843 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:53.112994 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:53.114209 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:53.114216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:53.114221 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:34:53.115116 (+ 895us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:53.115222 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:34:53.115264 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:53.115390 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:53.115513 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 17:34:53.115652 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:34:53.120520 24466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e51be117f014fe9b64f94de7e3815ec: Generated new TSK 0
I20260923 17:34:53.121446 24466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:34:53.129339 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 24388
I20260923 17:34:53.130774 24450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:34:53.221853 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 24388
2026-09-23T17:34:53Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1852 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:34:54 dist-test-slave-ppjl krb5kdc[24489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:34:54 dist-test-slave-ppjl krb5kdc[24489](info): set up 2 sockets
Sep 23 17:34:54 dist-test-slave-ppjl krb5kdc[24489](info): commencing operation
krb5kdc: starting...
W20260923 17:34:55.467952 21441 mini_kdc.cc:121] Time spent starting KDC: real 2.213s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:34:55 dist-test-slave-ppjl krb5kdc[24489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:34:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:34:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:34:55.659003 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:42137
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:35835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:42137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:55.979053 24502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:55.979426 24502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:55.979563 24502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:55.988163 24502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:34:55.988303 24502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:55.988395 24502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:55.988472 24502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:34:55.988531 24502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:34:55.998775 24502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:35835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:42137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:42137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:56.000226 24502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:56.002215 24502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:56.014479 24507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:56.014443 24508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:56.014513 24510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:56.015396 24502 server_base.cc:1034] running on GCE node
I20260923 17:34:56.016165 24502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:56.017674 24502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:56.018924 24502 hybrid_clock.cc:648] HybridClock initialized: now 1790184896018884 us; error 61 us; skew 500 ppm
I20260923 17:34:56.019367 24502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:56 dist-test-slave-ppjl krb5kdc[24489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184896, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:56.022607 24502 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:34:56.024233 24502 webserver.cc:460] Webserver started at http://127.20.240.126:36949/ using document root <none> and password file <none>
I20260923 17:34:56.024943 24502 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:56.025074 24502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:56.025381 24502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:56.028673 24502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "65e6d102623f49bd9fc3070865f6a2de"
format_stamp: "Formatted at 2026-09-23 17:34:56 on dist-test-slave-ppjl"
server_key: "9a0685d43c1e58859b8858f4fb3a4bd4"
server_key_iv: "3515be57530f1699b829a9ddee82eb1b"
server_key_version: "encryptionkey@0"
I20260923 17:34:56.029511 24502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "65e6d102623f49bd9fc3070865f6a2de"
format_stamp: "Formatted at 2026-09-23 17:34:56 on dist-test-slave-ppjl"
server_key: "9a0685d43c1e58859b8858f4fb3a4bd4"
server_key_iv: "3515be57530f1699b829a9ddee82eb1b"
server_key_version: "encryptionkey@0"
I20260923 17:34:56.035228 24502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:34:56.039067 24517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:56.040733 24502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:34:56.040948 24502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "65e6d102623f49bd9fc3070865f6a2de"
format_stamp: "Formatted at 2026-09-23 17:34:56 on dist-test-slave-ppjl"
server_key: "9a0685d43c1e58859b8858f4fb3a4bd4"
server_key_iv: "3515be57530f1699b829a9ddee82eb1b"
server_key_version: "encryptionkey@0"
I20260923 17:34:56.041148 24502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:56.059267 24502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:56.064332 24502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:56.064599 24502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:56.082723 24502 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:42137
I20260923 17:34:56.082744 24568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:42137 every 8 connection(s)
I20260923 17:34:56.084164 24502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:34:56.086910 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 24502
I20260923 17:34:56.087113 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:34:56.087921 21441 external_mini_cluster.cc:1428] Setting key b02caffe163472afb1a272ded11061fe
I20260923 17:34:56.088783 24569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:34:56 dist-test-slave-ppjl krb5kdc[24489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:56.101912 24569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de: Bootstrap starting.
I20260923 17:34:56.105242 24569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:56.106293 24569 log.cc:824] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:56.109243 24569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de: No bootstrap required, opened a new log
I20260923 17:34:56.109757 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:56.090123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60426 (local address 127.20.240.126:42137)
0923 17:34:56.090890 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:56.090903 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:56.090943 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:34:56.092528 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:56.092551 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:56.092979 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:56.093385 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:56.094799 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:56.096211 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:56.097462 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:56.098149 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:56.098340 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:56.102572 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:56.102582 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:56.102621 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:56.102670 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:56.105688 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:56.106416 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:56.106425 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:56.106434 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:56.106614 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:56.106993 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:56.106998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:56.107000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:56.107613 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:56.107910 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:56.108344 (+ 434us) server_negotiation.cc:299] Negotiation successful
0923 17:34:56.108758 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":332,"threads_started":1}
I20260923 17:34:56.116798 24569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e6d102623f49bd9fc3070865f6a2de" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42137 } }
I20260923 17:34:56.117273 24569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:56.117376 24569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e6d102623f49bd9fc3070865f6a2de, State: Initialized, Role: FOLLOWER
I20260923 17:34:56.117939 24569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e6d102623f49bd9fc3070865f6a2de" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42137 } }
I20260923 17:34:56.118208 24569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:34:56.118326 24569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:34:56.118450 24569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:56.120584 24569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e6d102623f49bd9fc3070865f6a2de" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42137 } }
I20260923 17:34:56.121064 24569 leader_election.cc:302] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65e6d102623f49bd9fc3070865f6a2de; no voters:
I20260923 17:34:56.121662 24569 leader_election.cc:288] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:34:56.121774 24574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:56.122885 24574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [term 1 LEADER]: Becoming Leader. State: Replica: 65e6d102623f49bd9fc3070865f6a2de, State: Running, Role: LEADER
I20260923 17:34:56.123432 24574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e6d102623f49bd9fc3070865f6a2de" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42137 } }
I20260923 17:34:56.123889 24569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:34:56.126332 24576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65e6d102623f49bd9fc3070865f6a2de. Latest consensus state: current_term: 1 leader_uuid: "65e6d102623f49bd9fc3070865f6a2de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e6d102623f49bd9fc3070865f6a2de" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42137 } } }
I20260923 17:34:56.126731 24576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:56.127352 24575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65e6d102623f49bd9fc3070865f6a2de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e6d102623f49bd9fc3070865f6a2de" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42137 } } }
I20260923 17:34:56.127650 24575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:56.127908 24583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:34:56.133673 24583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:34:56.147367 24583 catalog_manager.cc:1347] Generated new cluster ID: b73a3285ae824b829867122b54ba8fa1
I20260923 17:34:56.147572 24583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:34:56.157364 24583 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:34:56.158756 24583 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:34:56.168798 24583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65e6d102623f49bd9fc3070865f6a2de: Generated new TSK 0
I20260923 17:34:56.169684 24583 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:34:56.224017 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42137
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:35835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:56.554636 24595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:56.554932 24595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:56.555058 24595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:56.562454 24595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:56.562566 24595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:56.562718 24595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:34:56.572810 24595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:35835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:56.574112 24595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:56.575493 24595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:56.588449 24600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:56.588523 24601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:56.588498 24603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:56.589419 24595 server_base.cc:1034] running on GCE node
I20260923 17:34:56.590045 24595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:56.590975 24595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:56.592253 24595 hybrid_clock.cc:648] HybridClock initialized: now 1790184896592151 us; error 88 us; skew 500 ppm
I20260923 17:34:56.592608 24595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:56 dist-test-slave-ppjl krb5kdc[24489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184896, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:56.595913 24595 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:34:56.597458 24595 webserver.cc:460] Webserver started at http://127.20.240.65:42447/ using document root <none> and password file <none>
I20260923 17:34:56.598194 24595 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:56.598335 24595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:56.598628 24595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:56.601595 24595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "a37802addfcc454aabf9ccb08a19407c"
format_stamp: "Formatted at 2026-09-23 17:34:56 on dist-test-slave-ppjl"
server_key: "ad6a434576cd9a8b4056f4aef2e1a7e4"
server_key_iv: "e10dc7c07831f056a5ca16df67eaf9a5"
server_key_version: "encryptionkey@0"
I20260923 17:34:56.602370 24595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "a37802addfcc454aabf9ccb08a19407c"
format_stamp: "Formatted at 2026-09-23 17:34:56 on dist-test-slave-ppjl"
server_key: "ad6a434576cd9a8b4056f4aef2e1a7e4"
server_key_iv: "e10dc7c07831f056a5ca16df67eaf9a5"
server_key_version: "encryptionkey@0"
I20260923 17:34:56.607645 24595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260923 17:34:56.611301 24610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:56.612699 24595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:34:56.612859 24595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "a37802addfcc454aabf9ccb08a19407c"
format_stamp: "Formatted at 2026-09-23 17:34:56 on dist-test-slave-ppjl"
server_key: "ad6a434576cd9a8b4056f4aef2e1a7e4"
server_key_iv: "e10dc7c07831f056a5ca16df67eaf9a5"
server_key_version: "encryptionkey@0"
I20260923 17:34:56.613034 24595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:56.628480 24595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:56.632716 24595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:56.632978 24595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:56.634021 24595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:34:56.635560 24595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:56.635636 24595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:56.635723 24595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:56.635777 24595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:56.667551 24595 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:43515
I20260923 17:34:56.667625 24722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:43515 every 8 connection(s)
I20260923 17:34:56.669034 24595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:34:56.674152 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 24595
I20260923 17:34:56.674325 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:34:56.674947 21441 external_mini_cluster.cc:1428] Setting key 8740696f5ce7b0a16a7cde84d8cb8dce
Sep 23 17:34:56 dist-test-slave-ppjl krb5kdc[24489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184896, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:56.690076 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:56.671733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:41459 (local address 127.20.240.126:42137)
0923 17:34:56.672140 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:56.672146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:56.673162 (+ 1016us) server_negotiation.cc:407] Connection header received
0923 17:34:56.675182 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:56.675190 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:56.675281 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:56.675379 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:56.676948 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:56.677652 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:56.679355 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:56.679757 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:56.679873 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:56.683977 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:56.683983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:56.683987 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:56.684027 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:56.686591 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:56.687552 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:56.687557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:56.687560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:56.687711 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:56.688181 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:56.688185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:56.688187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:56.688483 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:56.688632 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:56.689440 (+ 808us) server_negotiation.cc:299] Negotiation successful
0923 17:34:56.689701 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":207,"threads_started":1}
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
I20260923 17:34:56.691231 24726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:56.672225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:42137 (local address 127.20.240.65:41459)
0923 17:34:56.672978 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:56.673024 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:56.674857 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:56.675639 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:56.675649 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:56.676178 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:56.676765 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:56.676784 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:56.677815 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:56.677819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:56.679148 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:56.679205 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:56.680004 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:56.680012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:56.680262 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:56.680971 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:56.680997 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:56.683680 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:56.686807 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:56.686816 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:56.686832 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:56.687412 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:56.687831 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:56.687835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:56.687837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:56.688048 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:56.688618 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:56.688627 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:56.689012 (+ 385us) client_negotiation.cc:769] Sending connection context
0923 17:34:56.689283 (+ 271us) client_negotiation.cc:240] Negotiation successful
0923 17:34:56.689912 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":273,"threads_started":1}
I20260923 17:34:56.693048 24723 heartbeater.cc:347] Connected to a master server at 127.20.240.126:42137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
I20260923 17:34:56.693486 24723 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:56.694402 24723 heartbeater.cc:511] Master 127.20.240.126:42137 requested a full tablet report, sending...
I20260923 17:34:56.696724 24534 ts_manager.cc:194] Registered new tserver with Master: a37802addfcc454aabf9ccb08a19407c (127.20.240.65:43515)
I20260923 17:34:56.698401 24534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:41459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:34:56.718313 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42137
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:35835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:57.042547 24729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:57.042887 24729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:57.043028 24729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:57.051236 24729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:57.051391 24729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:57.051556 24729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:34:57.061995 24729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:35835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:57.063385 24729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:57.064723 24729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:57.078114 24734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:57.078099 24735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:57.078087 24737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:57.078943 24729 server_base.cc:1034] running on GCE node
I20260923 17:34:57.079770 24729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:57.080957 24729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:57.082268 24729 hybrid_clock.cc:648] HybridClock initialized: now 1790184897082170 us; error 81 us; skew 500 ppm
I20260923 17:34:57.082752 24729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:57 dist-test-slave-ppjl krb5kdc[24489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184897, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:57.086283 24729 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:34:57.088135 24729 webserver.cc:460] Webserver started at http://127.20.240.66:43485/ using document root <none> and password file <none>
I20260923 17:34:57.089087 24729 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:57.089232 24729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:57.089579 24729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:57.093509 24729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "6badfe7b652f4b7cbb192428f2c2f11f"
format_stamp: "Formatted at 2026-09-23 17:34:57 on dist-test-slave-ppjl"
server_key: "e98526ad8cda3b5ce981aca33f54d9a2"
server_key_iv: "ef015619969cc1ae5f1c077180f8b7af"
server_key_version: "encryptionkey@0"
I20260923 17:34:57.094656 24729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "6badfe7b652f4b7cbb192428f2c2f11f"
format_stamp: "Formatted at 2026-09-23 17:34:57 on dist-test-slave-ppjl"
server_key: "e98526ad8cda3b5ce981aca33f54d9a2"
server_key_iv: "ef015619969cc1ae5f1c077180f8b7af"
server_key_version: "encryptionkey@0"
I20260923 17:34:57.101473 24729 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260923 17:34:57.105394 24744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:57.107030 24729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:34:57.107234 24729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "6badfe7b652f4b7cbb192428f2c2f11f"
format_stamp: "Formatted at 2026-09-23 17:34:57 on dist-test-slave-ppjl"
server_key: "e98526ad8cda3b5ce981aca33f54d9a2"
server_key_iv: "ef015619969cc1ae5f1c077180f8b7af"
server_key_version: "encryptionkey@0"
I20260923 17:34:57.107431 24729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:57.125295 24729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:57.130168 24729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:57.130450 24729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:57.131488 24729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:34:57.132824 24729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:57.132897 24729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:57.132992 24729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:57.133033 24729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:57.165022 24729 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:37785
I20260923 17:34:57.165051 24856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:37785 every 8 connection(s)
I20260923 17:34:57.166276 24729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:34:57.167061 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 24729
I20260923 17:34:57.167332 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:34:57.167920 21441 external_mini_cluster.cc:1428] Setting key c3af0c87a6f01176c3ab8689157ef388
Sep 23 17:34:57 dist-test-slave-ppjl krb5kdc[24489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184897, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
I20260923 17:34:57.189999 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:57.169210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:50739 (local address 127.20.240.126:42137)
0923 17:34:57.169399 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:57.169404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:57.170736 (+ 1332us) server_negotiation.cc:407] Connection header received
0923 17:34:57.173317 (+ 2581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:57.173323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:57.173399 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:57.173492 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:57.175673 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.176605 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.178293 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.178895 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.179027 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:57.182995 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:57.183004 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:57.183009 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:57.183060 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:57.186508 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:57.187465 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:57.187471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:57.187474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:57.187631 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:57.188129 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:57.188134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:57.188136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:57.188448 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:57.188600 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:57.189409 (+ 809us) server_negotiation.cc:299] Negotiation successful
0923 17:34:57.189646 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:34:57.190999 24859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:57.169719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:42137 (local address 127.20.240.66:50739)
0923 17:34:57.170536 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:57.170584 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:57.172934 (+ 2350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:57.173794 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:57.173806 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:57.174559 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:57.175435 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:57.175464 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.176796 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.176800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:57.178119 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:57.178132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.179122 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.179127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:57.179421 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:57.180117 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:57.180150 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:57.182677 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:57.186689 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:57.186697 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:57.186714 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:57.187289 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:57.187744 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:57.187747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:57.187766 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:57.187998 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:57.188579 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:57.188587 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:57.189024 (+ 437us) client_negotiation.cc:769] Sending connection context
0923 17:34:57.189272 (+ 248us) client_negotiation.cc:240] Negotiation successful
0923 17:34:57.189896 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":291,"threads_started":1}
I20260923 17:34:57.192466 24857 heartbeater.cc:347] Connected to a master server at 127.20.240.126:42137
I20260923 17:34:57.192757 24857 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:57.193511 24857 heartbeater.cc:511] Master 127.20.240.126:42137 requested a full tablet report, sending...
I20260923 17:34:57.194911 24534 ts_manager.cc:194] Registered new tserver with Master: 6badfe7b652f4b7cbb192428f2c2f11f (127.20.240.66:37785)
I20260923 17:34:57.195765 24534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:50739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:34:57.212509 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42137
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:35835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:57.525102 24862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:57.525408 24862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:57.525503 24862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:57.532984 24862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:57.533102 24862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:57.533215 24862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:34:57.543126 24862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:35835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:57.544485 24862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:57.545843 24862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:57.559059 24867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:57.559108 24868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:57.559484 24870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:57.560055 24862 server_base.cc:1034] running on GCE node
I20260923 17:34:57.560631 24862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:57.561581 24862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:57.562785 24862 hybrid_clock.cc:648] HybridClock initialized: now 1790184897562731 us; error 38 us; skew 500 ppm
I20260923 17:34:57.563118 24862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:57 dist-test-slave-ppjl krb5kdc[24489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184897, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:57.566251 24862 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:34:57.567665 24862 webserver.cc:460] Webserver started at http://127.20.240.67:45001/ using document root <none> and password file <none>
I20260923 17:34:57.568370 24862 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:57.568485 24862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:57.568785 24862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:57.571851 24862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "a33b4847fe7e47e49138b48674658965"
format_stamp: "Formatted at 2026-09-23 17:34:57 on dist-test-slave-ppjl"
server_key: "762c9a75d0c08c4f230d2662f76433af"
server_key_iv: "deceda77330217ff91dfd52f0acdfe01"
server_key_version: "encryptionkey@0"
I20260923 17:34:57.572629 24862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "a33b4847fe7e47e49138b48674658965"
format_stamp: "Formatted at 2026-09-23 17:34:57 on dist-test-slave-ppjl"
server_key: "762c9a75d0c08c4f230d2662f76433af"
server_key_iv: "deceda77330217ff91dfd52f0acdfe01"
server_key_version: "encryptionkey@0"
I20260923 17:34:57.577908 24862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:34:57.581743 24877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:57.583364 24862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:34:57.583532 24862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "a33b4847fe7e47e49138b48674658965"
format_stamp: "Formatted at 2026-09-23 17:34:57 on dist-test-slave-ppjl"
server_key: "762c9a75d0c08c4f230d2662f76433af"
server_key_iv: "deceda77330217ff91dfd52f0acdfe01"
server_key_version: "encryptionkey@0"
I20260923 17:34:57.583719 24862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:57.601576 24862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:57.605839 24862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:57.606135 24862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:57.607066 24862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:34:57.608419 24862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:34:57.608495 24862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:57.608618 24862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:34:57.608670 24862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:57.639780 24862 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:46101
I20260923 17:34:57.639814 24989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:46101 every 8 connection(s)
I20260923 17:34:57.641167 24862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:34:57.650449 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 24862
I20260923 17:34:57.650602 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:34:57.651031 21441 external_mini_cluster.cc:1428] Setting key 5c06b05ffaeaa66509270c48dd4e1985
Sep 23 17:34:57 dist-test-slave-ppjl krb5kdc[24489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184897, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:57.660841 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:57.643784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:37063 (local address 127.20.240.126:42137)
0923 17:34:57.643962 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:57.643968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:57.645217 (+ 1249us) server_negotiation.cc:407] Connection header received
0923 17:34:57.647062 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:57.647067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:57.647140 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:57.647264 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:57.648772 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.649525 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.651206 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.651752 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.651878 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:57.655277 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:57.655283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:57.655286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:57.655320 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:57.657838 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:57.658545 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:57.658551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:57.658553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:57.658679 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:57.659077 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:57.659082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:57.659086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:57.659406 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:57.659549 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:57.660261 (+ 712us) server_negotiation.cc:299] Negotiation successful
0923 17:34:57.660494 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:34:57.661839 24992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:57.644315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:42137 (local address 127.20.240.67:37063)
0923 17:34:57.645056 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0923 17:34:57.645103 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:34:57.646795 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:34:57.647529 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:34:57.647537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:34:57.648050 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:34:57.648613 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:57.648631 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.649670 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.649674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:57.651013 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:34:57.651025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.651993 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.652000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:34:57.652246 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:57.652884 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:34:57.652912 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:34:57.655050 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:34:57.657977 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:57.657986 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:57.658006 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:57.658428 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:57.658782 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:34:57.658787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:34:57.658790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:34:57.658975 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:34:57.659541 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:34:57.659549 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:34:57.659901 (+ 352us) client_negotiation.cc:769] Sending connection context
0923 17:34:57.660150 (+ 249us) client_negotiation.cc:240] Negotiation successful
0923 17:34:57.660730 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":260,"threads_started":1}
I20260923 17:34:57.663388 24990 heartbeater.cc:347] Connected to a master server at 127.20.240.126:42137
I20260923 17:34:57.663681 24990 heartbeater.cc:464] Registering TS with master...
I20260923 17:34:57.664420 24990 heartbeater.cc:511] Master 127.20.240.126:42137 requested a full tablet report, sending...
I20260923 17:34:57.666088 24534 ts_manager.cc:194] Registered new tserver with Master: a33b4847fe7e47e49138b48674658965 (127.20.240.67:46101)
I20260923 17:34:57.666733 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:34:57.667150 24534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:37063
Sep 23 17:34:57 dist-test-slave-ppjl krb5kdc[24489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184897, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260923 17:34:57.701643 24723 heartbeater.cc:503] Master 127.20.240.126:42137 was elected leader, sending a full tablet report...
Sep 23 17:34:57 dist-test-slave-ppjl krb5kdc[24489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184897, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:57.713306 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:57.698740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60442 (local address 127.20.240.126:42137)
0923 17:34:57.698994 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:57.699001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:57.699417 (+ 416us) server_negotiation.cc:407] Connection header received
0923 17:34:57.699583 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:57.699588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:57.699657 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:57.699732 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:57.701155 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.701979 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.703646 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.704046 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.704141 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:57.707957 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:57.707963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:57.707966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:57.708002 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:57.710408 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:57.711177 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:57.711184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:57.711188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:57.711368 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:57.711848 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:57.711851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:57.711853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:57.712243 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:57.712457 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:57.712789 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 17:34:57.712967 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260923 17:34:57.713879 24534 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60442
I20260923 17:34:57.715363 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 24595
Sep 23 17:34:57 dist-test-slave-ppjl krb5kdc[24489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184897, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:34:57.731353 25001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:57.716401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.20.240.65:43515)
0923 17:34:57.716969 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:57.716976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:57.717001 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:34:57.717062 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:57.717071 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:57.717307 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:57.717447 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:57.718733 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.719776 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.721105 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.721687 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.721789 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:57.725817 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:57.725832 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:57.725847 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:57.725893 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:57.728750 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:57.729487 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:57.729494 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:57.729496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:57.729612 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:57.729955 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:57.729958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:57.729960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:57.730305 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:57.730529 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:57.730817 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 17:34:57.731018 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":259,"threads_started":1}
W20260923 17:34:57.732604 24718 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51664
W20260923 17:34:57.733705 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 24595: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:34:57.733920 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 24595
I20260923 17:34:57.746662 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 24729
Sep 23 17:34:57 dist-test-slave-ppjl krb5kdc[24489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184897, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:34:57.763867 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:57.747579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.20.240.66:37785)
0923 17:34:57.748132 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:57.748139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:57.748167 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:34:57.748236 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:57.748243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:57.748450 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:57.748584 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:57.749793 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.750775 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.752380 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.752987 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.753112 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:57.757836 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:57.757851 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:57.757867 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:57.757910 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:57.760775 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:57.761513 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:57.761521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:57.761523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:57.761678 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:57.762086 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:57.762090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:57.762092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:57.762481 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:57.762821 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:57.763196 (+ 375us) server_negotiation.cc:299] Negotiation successful
0923 17:34:57.763478 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":271,"threads_started":1}
W20260923 17:34:57.765167 24852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38166
W20260923 17:34:57.766503 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 24729: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:34:57.766661 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 24729
I20260923 17:34:57.778708 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 24862
Sep 23 17:34:57 dist-test-slave-ppjl krb5kdc[24489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184897, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:34:57.795833 25003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:57.779725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51830 (local address 127.20.240.67:46101)
0923 17:34:57.780417 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:57.780425 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:57.780454 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:34:57.780520 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:57.780527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:57.780743 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:57.780895 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:57.782289 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.783465 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.784777 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:57.785495 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:57.785607 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:57.790170 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:57.790185 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:57.790201 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:57.790248 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:57.793027 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:57.793772 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:57.793780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:57.793783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:57.793912 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:57.794241 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:57.794244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:57.794246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:57.794613 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:57.794837 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:57.795141 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 17:34:57.795477 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":311,"threads_started":1}
W20260923 17:34:57.797101 24985 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51830
W20260923 17:34:57.798144 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 24862: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:34:57.798300 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 24862
I20260923 17:34:57.810770 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 24502
I20260923 17:34:57.811977 24564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:34:57.914366 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 24502
2026-09-23T17:34:57Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4698 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:34:58 dist-test-slave-ppjl krb5kdc[25007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:34:58 dist-test-slave-ppjl krb5kdc[25007](info): set up 2 sockets
Sep 23 17:34:58 dist-test-slave-ppjl krb5kdc[25007](info): commencing operation
krb5kdc: starting...
W20260923 17:34:58.932497 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:34:59 dist-test-slave-ppjl krb5kdc[25007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:34:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:34:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:34:59.135900 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:40453
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:40453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:34:59.457886 25020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:34:59.458228 25020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:34:59.458374 25020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:34:59.466035 25020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:34:59.466173 25020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:34:59.466261 25020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:34:59.466336 25020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:34:59.466399 25020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:34:59.476679 25020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:40453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:40453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:34:59.478088 25020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:34:59.479461 25020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:34:59.491020 25025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:59.491053 25026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:34:59.491333 25028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:34:59.491734 25020 server_base.cc:1034] running on GCE node
I20260923 17:34:59.492524 25020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:34:59.493968 25020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:34:59.495203 25020 hybrid_clock.cc:648] HybridClock initialized: now 1790184899495126 us; error 39 us; skew 500 ppm
I20260923 17:34:59.495543 25020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:34:59 dist-test-slave-ppjl krb5kdc[25007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184899, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:34:59.498446 25020 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:34:59.499938 25020 webserver.cc:460] Webserver started at http://127.20.240.126:35647/ using document root <none> and password file <none>
I20260923 17:34:59.500583 25020 fs_manager.cc:359] Metadata directory not provided
I20260923 17:34:59.500679 25020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:34:59.500963 25020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:34:59.503904 25020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "3665a8943da64be09566e2fce71d7f34"
format_stamp: "Formatted at 2026-09-23 17:34:59 on dist-test-slave-ppjl"
server_key: "0f0976f0fa8150cf61155293277aef17"
server_key_iv: "714313f844b59167ed5c4ac6cba60b02"
server_key_version: "encryptionkey@0"
I20260923 17:34:59.504640 25020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "3665a8943da64be09566e2fce71d7f34"
format_stamp: "Formatted at 2026-09-23 17:34:59 on dist-test-slave-ppjl"
server_key: "0f0976f0fa8150cf61155293277aef17"
server_key_iv: "714313f844b59167ed5c4ac6cba60b02"
server_key_version: "encryptionkey@0"
I20260923 17:34:59.509840 25020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260923 17:34:59.513458 25035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:34:59.514798 25020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:34:59.514951 25020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "3665a8943da64be09566e2fce71d7f34"
format_stamp: "Formatted at 2026-09-23 17:34:59 on dist-test-slave-ppjl"
server_key: "0f0976f0fa8150cf61155293277aef17"
server_key_iv: "714313f844b59167ed5c4ac6cba60b02"
server_key_version: "encryptionkey@0"
I20260923 17:34:59.515183 25020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:34:59.526889 25020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:34:59.530961 25020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:34:59.531275 25020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:34:59.547785 25020 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:40453
I20260923 17:34:59.547789 25086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:40453 every 8 connection(s)
I20260923 17:34:59.549264 25020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:34:59.553258 25087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:34:59.554211 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 25020
I20260923 17:34:59.554415 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:34:59.555054 21441 external_mini_cluster.cc:1428] Setting key 25235cdad0ab7ae54b3f78b90d50c53d
I20260923 17:34:59.566841 25087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34: Bootstrap starting.
Sep 23 17:34:59 dist-test-slave-ppjl krb5kdc[25007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:34:59.570360 25087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34: Neither blocks nor log segments found. Creating new log.
I20260923 17:34:59.571754 25087 log.cc:824] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34: Log is configured to *not* fsync() on all Append() calls
I20260923 17:34:59.575395 25087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34: No bootstrap required, opened a new log
I20260923 17:34:59.579051 25090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:34:59.557263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.20.240.126:40453)
0923 17:34:59.558196 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0923 17:34:59.558213 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:34:59.558266 (+ 53us) server_negotiation.cc:407] Connection header received
0923 17:34:59.560145 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:34:59.560176 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:34:59.560700 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:34:59.561142 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:34:59.562709 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:59.564085 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:59.565384 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:34:59.566078 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:34:59.566254 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:34:59.570449 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:34:59.570462 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:34:59.570483 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:34:59.570539 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:34:59.574289 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:59.575044 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:59.575053 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:59.575061 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:59.575340 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:34:59.575794 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:34:59.575799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:34:59.575803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:34:59.576553 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:34:59.576861 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 17:34:59.577199 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 17:34:59.577677 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":401,"threads_started":1}
I20260923 17:34:59.584776 25087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3665a8943da64be09566e2fce71d7f34" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40453 } }
I20260923 17:34:59.585116 25087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:34:59.585191 25087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3665a8943da64be09566e2fce71d7f34, State: Initialized, Role: FOLLOWER
I20260923 17:34:59.585667 25087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3665a8943da64be09566e2fce71d7f34" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40453 } }
I20260923 17:34:59.585860 25087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:34:59.585949 25087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:34:59.586066 25087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:34:59.588346 25087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3665a8943da64be09566e2fce71d7f34" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40453 } }
I20260923 17:34:59.588780 25087 leader_election.cc:302] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3665a8943da64be09566e2fce71d7f34; no voters:
I20260923 17:34:59.589323 25087 leader_election.cc:288] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:34:59.589452 25092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:34:59.590075 25092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [term 1 LEADER]: Becoming Leader. State: Replica: 3665a8943da64be09566e2fce71d7f34, State: Running, Role: LEADER
I20260923 17:34:59.590575 25092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3665a8943da64be09566e2fce71d7f34" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40453 } }
I20260923 17:34:59.591434 25087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:34:59.593497 25093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3665a8943da64be09566e2fce71d7f34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3665a8943da64be09566e2fce71d7f34" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40453 } } }
I20260923 17:34:59.593452 25094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3665a8943da64be09566e2fce71d7f34. Latest consensus state: current_term: 1 leader_uuid: "3665a8943da64be09566e2fce71d7f34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3665a8943da64be09566e2fce71d7f34" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 40453 } } }
I20260923 17:34:59.593896 25093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:59.593916 25094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34 [sys.catalog]: This master's current role is: LEADER
I20260923 17:34:59.594937 25098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:34:59.601104 25098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:34:59.610414 25098 catalog_manager.cc:1347] Generated new cluster ID: 3e968916c6b94c5eac4149d03822094c
I20260923 17:34:59.610538 25098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:34:59.630743 25098 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:34:59.632172 25098 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:34:59.644806 25098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3665a8943da64be09566e2fce71d7f34: Generated new TSK 0
I20260923 17:34:59.645758 25098 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:34:59.702896 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:40453
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:00.046304 25113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:00.046591 25113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:00.046707 25113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:00.054143 25113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:00.054268 25113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:00.054422 25113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:35:00.065265 25113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:40453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:00.066659 25113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:00.068101 25113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:00.081477 25118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:00.081907 25119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:00.081961 25121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:00.082269 25113 server_base.cc:1034] running on GCE node
I20260923 17:35:00.082926 25113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:00.084024 25113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:00.085249 25113 hybrid_clock.cc:648] HybridClock initialized: now 1790184900085228 us; error 76 us; skew 500 ppm
I20260923 17:35:00.085590 25113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:00 dist-test-slave-ppjl krb5kdc[25007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184900, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:00.088985 25113 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:35:00.090731 25113 webserver.cc:460] Webserver started at http://127.20.240.65:42961/ using document root <none> and password file <none>
I20260923 17:35:00.091518 25113 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:00.091645 25113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:00.091938 25113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:00.095014 25113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "fd3f7bf39f514c1994454a17a37d43e4"
format_stamp: "Formatted at 2026-09-23 17:35:00 on dist-test-slave-ppjl"
server_key: "5a9f1f0009bac8b63983277df073bc63"
server_key_iv: "8a988f816271c4559503d41d4384dfe7"
server_key_version: "encryptionkey@0"
I20260923 17:35:00.095824 25113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "fd3f7bf39f514c1994454a17a37d43e4"
format_stamp: "Formatted at 2026-09-23 17:35:00 on dist-test-slave-ppjl"
server_key: "5a9f1f0009bac8b63983277df073bc63"
server_key_iv: "8a988f816271c4559503d41d4384dfe7"
server_key_version: "encryptionkey@0"
I20260923 17:35:00.101090 25113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:35:00.104836 25128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:00.106273 25113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:35:00.106427 25113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "fd3f7bf39f514c1994454a17a37d43e4"
format_stamp: "Formatted at 2026-09-23 17:35:00 on dist-test-slave-ppjl"
server_key: "5a9f1f0009bac8b63983277df073bc63"
server_key_iv: "8a988f816271c4559503d41d4384dfe7"
server_key_version: "encryptionkey@0"
I20260923 17:35:00.106637 25113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:00.126206 25113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:00.130721 25113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:00.130976 25113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:00.132006 25113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:00.133411 25113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:00.133488 25113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:00.133581 25113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:00.133623 25113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:00.169080 25113 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:45499
I20260923 17:35:00.169119 25240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:45499 every 8 connection(s)
I20260923 17:35:00.170338 25113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:35:00.176683 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 25113
I20260923 17:35:00.176888 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:35:00.177484 21441 external_mini_cluster.cc:1428] Setting key 70b5352a2390e29c13a90d57da599649
Sep 23 17:35:00 dist-test-slave-ppjl krb5kdc[25007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184900, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:35:00.190896 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:00.172932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:33701 (local address 127.20.240.126:40453)
0923 17:35:00.173334 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:00.173339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:00.174380 (+ 1041us) server_negotiation.cc:407] Connection header received
0923 17:35:00.176262 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:00.176268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:00.176352 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:00.176448 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:00.178087 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:00.178817 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:00.180434 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:00.180846 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:00.180959 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:00.184686 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:00.184693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:00.184698 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:00.184736 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:00.187214 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:00.188228 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:00.188233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:00.188235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:00.188372 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:00.188910 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:00.188914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:00.188916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:00.189297 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:00.189426 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:00.190440 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0923 17:35:00.190628 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":184,"threads_started":1}
I20260923 17:35:00.192328 25244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:00.173505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:40453 (local address 127.20.240.65:33701)
0923 17:35:00.174216 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:00.174254 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:00.175974 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:00.176707 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:00.176716 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:00.177281 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:00.177906 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:00.177926 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:00.178988 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:00.178992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:00.180284 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:00.180298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:00.181074 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:00.181082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:00.181314 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:00.182038 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:00.182066 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:00.184429 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:00.187378 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:00.187387 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:00.187407 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:00.187988 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:00.188488 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:00.188492 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:00.188496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:00.188757 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:00.189431 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:00.189439 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:00.189909 (+ 470us) client_negotiation.cc:769] Sending connection context
0923 17:35:00.190219 (+ 310us) client_negotiation.cc:240] Negotiation successful
0923 17:35:00.190823 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":275,"threads_started":1}
I20260923 17:35:00.193924 25241 heartbeater.cc:347] Connected to a master server at 127.20.240.126:40453
I20260923 17:35:00.194233 25241 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
I20260923 17:35:00.195024 25241 heartbeater.cc:511] Master 127.20.240.126:40453 requested a full tablet report, sending...
I20260923 17:35:00.197368 25052 ts_manager.cc:194] Registered new tserver with Master: fd3f7bf39f514c1994454a17a37d43e4 (127.20.240.65:45499)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
I20260923 17:35:00.198942 25052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:33701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:35:00.220718 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:40453
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:00.543812 25247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:00.544137 25247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:00.544255 25247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:00.554543 25247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:00.554730 25247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:00.554934 25247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:35:00.571301 25247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:40453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:00.572727 25247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:00.574026 25247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:00.587226 25252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:00.587323 25253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:00.587294 25255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:00.588100 25247 server_base.cc:1034] running on GCE node
I20260923 17:35:00.588686 25247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:00.589607 25247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:00.590799 25247 hybrid_clock.cc:648] HybridClock initialized: now 1790184900590742 us; error 52 us; skew 500 ppm
I20260923 17:35:00.591095 25247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:00 dist-test-slave-ppjl krb5kdc[25007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184900, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:00.594182 25247 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:35:00.595809 25247 webserver.cc:460] Webserver started at http://127.20.240.66:42583/ using document root <none> and password file <none>
I20260923 17:35:00.596550 25247 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:00.596668 25247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:00.596956 25247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:00.599999 25247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "e90cf9eaf1de4d409905f8dd1652d570"
format_stamp: "Formatted at 2026-09-23 17:35:00 on dist-test-slave-ppjl"
server_key: "2694b0d5e5ad90c10b916bba6dc3a0b9"
server_key_iv: "568310f76645717f3664d9aa898f0c9a"
server_key_version: "encryptionkey@0"
I20260923 17:35:00.600756 25247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "e90cf9eaf1de4d409905f8dd1652d570"
format_stamp: "Formatted at 2026-09-23 17:35:00 on dist-test-slave-ppjl"
server_key: "2694b0d5e5ad90c10b916bba6dc3a0b9"
server_key_iv: "568310f76645717f3664d9aa898f0c9a"
server_key_version: "encryptionkey@0"
I20260923 17:35:00.606027 25247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:35:00.609534 25262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:00.611093 25247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:35:00.611316 25247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "e90cf9eaf1de4d409905f8dd1652d570"
format_stamp: "Formatted at 2026-09-23 17:35:00 on dist-test-slave-ppjl"
server_key: "2694b0d5e5ad90c10b916bba6dc3a0b9"
server_key_iv: "568310f76645717f3664d9aa898f0c9a"
server_key_version: "encryptionkey@0"
I20260923 17:35:00.611545 25247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:00.627817 25247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:00.632356 25247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:00.632635 25247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:00.633687 25247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:00.635087 25247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:00.635202 25247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:00.635324 25247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:00.635380 25247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:00.667908 25247 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:41451
I20260923 17:35:00.667958 25374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:41451 every 8 connection(s)
I20260923 17:35:00.669209 25247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:35:00.679585 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 25247
I20260923 17:35:00.679801 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:35:00.680349 21441 external_mini_cluster.cc:1428] Setting key 0cbe9affcf87baeb21bb419047e98a93
Sep 23 17:35:00 dist-test-slave-ppjl krb5kdc[25007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184900, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:35:00.691835 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:00.671985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:58201 (local address 127.20.240.126:40453)
0923 17:35:00.672244 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:00.672248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:00.673510 (+ 1262us) server_negotiation.cc:407] Connection header received
0923 17:35:00.675447 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:00.675453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:00.675543 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:00.675642 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:00.677474 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:00.678210 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:00.680328 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:00.680823 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:00.681000 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:00.685428 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:00.685434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:00.685437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:00.685478 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:00.688374 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:00.689230 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:00.689236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:00.689240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:00.689407 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:00.689865 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:00.689870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:00.689872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:00.690212 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:00.690361 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:00.691246 (+ 885us) server_negotiation.cc:299] Negotiation successful
0923 17:35:00.691496 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:35:00.692876 25377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:00.672493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:40453 (local address 127.20.240.66:58201)
0923 17:35:00.673324 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:00.673369 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:00.675107 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:00.675918 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:00.675927 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:00.676558 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:00.677266 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:00.677290 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:00.678397 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:00.678401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:00.680128 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:00.680146 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:00.681088 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:00.681093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:00.681435 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:00.682135 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:00.682169 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:00.685062 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:00.688571 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:00.688579 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:00.688596 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:00.689069 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:00.689528 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:00.689531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:00.689534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:00.689759 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:00.690356 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:00.690363 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:00.690810 (+ 447us) client_negotiation.cc:769] Sending connection context
0923 17:35:00.691075 (+ 265us) client_negotiation.cc:240] Negotiation successful
0923 17:35:00.691597 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":285,"threads_started":1}
I20260923 17:35:00.694643 25375 heartbeater.cc:347] Connected to a master server at 127.20.240.126:40453
I20260923 17:35:00.695001 25375 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:00.695935 25375 heartbeater.cc:511] Master 127.20.240.126:40453 requested a full tablet report, sending...
I20260923 17:35:00.697849 25052 ts_manager.cc:194] Registered new tserver with Master: e90cf9eaf1de4d409905f8dd1652d570 (127.20.240.66:41451)
I20260923 17:35:00.699050 25052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:58201
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:35:00.728456 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:40453
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:01.053833 25380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:01.054224 25380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:01.054418 25380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:01.062371 25380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:01.062541 25380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:01.062709 25380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:35:01.073186 25380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:40453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:01.074628 25380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:01.076251 25380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:01.090807 25385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:01.090842 25388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:01.090773 25386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:01.091672 25380 server_base.cc:1034] running on GCE node
I20260923 17:35:01.092370 25380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:01.093436 25380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:01.094676 25380 hybrid_clock.cc:648] HybridClock initialized: now 1790184901094601 us; error 64 us; skew 500 ppm
I20260923 17:35:01.095021 25380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:01 dist-test-slave-ppjl krb5kdc[25007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184901, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:01.098618 25380 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:35:01.100245 25380 webserver.cc:460] Webserver started at http://127.20.240.67:33835/ using document root <none> and password file <none>
I20260923 17:35:01.100976 25380 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:01.101094 25380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:01.101375 25380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:01.104413 25380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "f655f425196c400a9e82f6bd916abb38"
format_stamp: "Formatted at 2026-09-23 17:35:01 on dist-test-slave-ppjl"
server_key: "a98e1081d8fae1f9300f0773ee3ed9ba"
server_key_iv: "48472380058d1b6aa90f8585db61a8e8"
server_key_version: "encryptionkey@0"
I20260923 17:35:01.105186 25380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "f655f425196c400a9e82f6bd916abb38"
format_stamp: "Formatted at 2026-09-23 17:35:01 on dist-test-slave-ppjl"
server_key: "a98e1081d8fae1f9300f0773ee3ed9ba"
server_key_iv: "48472380058d1b6aa90f8585db61a8e8"
server_key_version: "encryptionkey@0"
I20260923 17:35:01.110671 25380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:35:01.114463 25395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:01.115995 25380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 17:35:01.116146 25380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "f655f425196c400a9e82f6bd916abb38"
format_stamp: "Formatted at 2026-09-23 17:35:01 on dist-test-slave-ppjl"
server_key: "a98e1081d8fae1f9300f0773ee3ed9ba"
server_key_iv: "48472380058d1b6aa90f8585db61a8e8"
server_key_version: "encryptionkey@0"
I20260923 17:35:01.116376 25380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:01.142000 25380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:01.146302 25380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:01.146590 25380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:01.147609 25380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:01.149000 25380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:01.149070 25380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:01.149168 25380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:01.149225 25380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:01.181165 25380 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:39017
I20260923 17:35:01.181226 25507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:39017 every 8 connection(s)
I20260923 17:35:01.182425 25380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:35:01.187633 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 25380
I20260923 17:35:01.187811 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:35:01.188247 21441 external_mini_cluster.cc:1428] Setting key 83a43aabf2d0cbd31a252d59c414f390
Sep 23 17:35:01 dist-test-slave-ppjl krb5kdc[25007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184901, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:35:01.201921 25241 heartbeater.cc:503] Master 127.20.240.126:40453 was elected leader, sending a full tablet report...
I20260923 17:35:01.204192 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:01.185257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:33153 (local address 127.20.240.126:40453)
0923 17:35:01.185447 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:01.185452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:01.186600 (+ 1148us) server_negotiation.cc:407] Connection header received
0923 17:35:01.188727 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:01.188734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:01.188840 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:01.188940 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:01.190693 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:01.191762 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:01.193461 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:01.193877 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:01.193988 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:01.197579 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:01.197585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:01.197588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:01.197630 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:01.200343 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:01.201309 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:01.201315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:01.201318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:01.201481 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:01.202067 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:01.202134 (+ 67us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:01.202138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:01.202554 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:01.202723 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:01.203639 (+ 916us) server_negotiation.cc:299] Negotiation successful
0923 17:35:01.203890 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:35:01.205195 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:01.185549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:40453 (local address 127.20.240.67:33153)
0923 17:35:01.186380 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:01.186427 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:01.188386 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:01.189269 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:01.189279 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:01.189854 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:01.190498 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:01.190519 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:01.191922 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:01.191927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:01.193306 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:01.193320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:01.194097 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:01.194104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:01.194347 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:01.194972 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:01.194995 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:01.197351 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:01.200507 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:01.200514 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:01.200528 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:01.201130 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:01.201641 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:01.201646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:01.201650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:01.201929 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:01.202700 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:01.202708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:01.203185 (+ 477us) client_negotiation.cc:769] Sending connection context
0923 17:35:01.203520 (+ 335us) client_negotiation.cc:240] Negotiation successful
0923 17:35:01.204063 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":285,"threads_started":1}
I20260923 17:35:01.206833 25508 heartbeater.cc:347] Connected to a master server at 127.20.240.126:40453
I20260923 17:35:01.207144 25508 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:01.207927 25508 heartbeater.cc:511] Master 127.20.240.126:40453 requested a full tablet report, sending...
I20260923 17:35:01.209442 25051 ts_manager.cc:194] Registered new tserver with Master: f655f425196c400a9e82f6bd916abb38 (127.20.240.67:39017)
I20260923 17:35:01.210326 25051 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:33153
I20260923 17:35:01.216445 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:35:01 dist-test-slave-ppjl krb5kdc[25007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:35:01 dist-test-slave-ppjl krb5kdc[25007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:35:01.262872 25515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:01.247417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60814 (local address 127.20.240.65:45499)
0923 17:35:01.248052 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:01.248059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:01.248087 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:35:01.248213 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:01.248221 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:01.248436 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:01.248567 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:01.249965 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:01.251019 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:01.252401 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:01.252986 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:01.253085 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:01.257160 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:01.257176 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:01.257199 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:01.257284 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:01.260141 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:01.260851 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:01.260857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:01.260860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:01.260981 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:01.261393 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:01.261397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:01.261399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:01.261749 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:01.261964 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:01.262294 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 17:35:01.262512 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":339,"threads_started":1}
Sep 23 17:35:01 dist-test-slave-ppjl krb5kdc[25007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:35:01.280797 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:01.267313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.20.240.126:40453)
0923 17:35:01.267523 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:01.267529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:01.267651 (+ 122us) server_negotiation.cc:407] Connection header received
0923 17:35:01.267807 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:01.267813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:01.267875 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:01.267955 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:01.269366 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:01.270096 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:01.271462 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:01.271858 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:01.271953 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:01.275715 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:01.275720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:01.275724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:01.275761 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:01.278333 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:01.278958 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:01.278962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:01.278965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:01.279092 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:01.279496 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:01.279499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:01.279501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:01.279824 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:01.280012 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:01.280306 (+ 294us) server_negotiation.cc:299] Negotiation successful
0923 17:35:01.280491 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 17:35:01.281399 25051 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45416
I20260923 17:35:01.282384 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 25113
I20260923 17:35:01.294886 25515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:01.283235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.20.240.65:45499)
0923 17:35:01.283468 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:01.283474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:01.283606 (+ 132us) server_negotiation.cc:407] Connection header received
0923 17:35:01.283732 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:01.283735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:01.283831 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:01.283908 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:01.285220 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:01.285885 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:01.287230 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:01.287640 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:01.287734 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:01.290133 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:01.290138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:01.290142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:01.290174 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:01.292681 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:01.293249 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:01.293253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:01.293255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:01.293389 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:01.293678 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:01.293681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:01.293683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:01.293921 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:01.294100 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:01.294437 (+ 337us) server_negotiation.cc:299] Negotiation successful
0923 17:35:01.294594 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:35:01.295449 25236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:01.378298 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 25113
I20260923 17:35:01.391276 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 25247
Sep 23 17:35:01 dist-test-slave-ppjl krb5kdc[25007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:35:01.407433 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:01.392164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.20.240.66:41451)
0923 17:35:01.392763 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:01.392772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:01.392805 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:35:01.392868 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:01.392874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:01.393126 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:01.393267 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:01.394449 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:01.395709 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:01.397174 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:01.397747 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:01.397868 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:01.401832 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:01.401845 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:01.401861 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:01.401900 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:01.404631 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:01.405289 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:01.405297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:01.405299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:01.405455 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:01.405820 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:01.405824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:01.405826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:01.406178 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:01.406489 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:01.406776 (+ 287us) server_negotiation.cc:299] Negotiation successful
0923 17:35:01.407038 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":328,"threads_started":1}
I20260923 17:35:01.408682 25370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:01.490563 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 25247
I20260923 17:35:01.503541 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 25380
Sep 23 17:35:01 dist-test-slave-ppjl krb5kdc[25007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:35:01.520918 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:01.504568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37796 (local address 127.20.240.67:39017)
0923 17:35:01.505263 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:01.505270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:01.505298 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:35:01.505359 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:01.505365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:01.505567 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:01.505733 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:01.507062 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:01.508115 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:01.509419 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:01.510017 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:01.510139 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:01.514747 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:01.514762 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:01.514779 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:01.514819 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:01.517895 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:01.518715 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:01.518723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:01.518725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:01.518855 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:01.519242 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:01.519247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:01.519251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:01.519743 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:01.520011 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:01.520307 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 17:35:01.520582 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":374,"threads_started":1}
I20260923 17:35:01.522327 25503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:01.606369 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 25380
I20260923 17:35:01.618949 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 25020
I20260923 17:35:01.620452 25082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:01.725793 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 25020
2026-09-23T17:35:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3810 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:35:01 dist-test-slave-ppjl krb5kdc[25527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:35:01 dist-test-slave-ppjl krb5kdc[25527](info): set up 2 sockets
Sep 23 17:35:01 dist-test-slave-ppjl krb5kdc[25527](info): commencing operation
krb5kdc: starting...
W20260923 17:35:02.732064 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:35:02 dist-test-slave-ppjl krb5kdc[25527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:35:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:35:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:35:02.928386 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:33339
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:33339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:03.248349 25540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:03.248677 25540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:03.248816 25540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:03.256593 25540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:35:03.256747 25540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:03.256829 25540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:03.256913 25540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:35:03.256978 25540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:35:03.267470 25540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:33339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:33339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:03.268867 25540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:03.270231 25540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:03.282205 25545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:03.282099 25546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:03.282136 25548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:03.282897 25540 server_base.cc:1034] running on GCE node
I20260923 17:35:03.283806 25540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:03.285374 25540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:03.286612 25540 hybrid_clock.cc:648] HybridClock initialized: now 1790184903286568 us; error 47 us; skew 500 ppm
I20260923 17:35:03.286962 25540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:03 dist-test-slave-ppjl krb5kdc[25527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184903, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:03.295482 25540 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:35:03.296810 25540 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:03.296911 25540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:03.297359 25540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:03.300562 25540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "8f5c713d883443928935e351ba8b0be2"
format_stamp: "Formatted at 2026-09-23 17:35:03 on dist-test-slave-ppjl"
server_key: "64f9057dad3ac3dc4be15066585a84f6"
server_key_iv: "dd008d3362309bf0f5e089e45b47e2da"
server_key_version: "encryptionkey@0"
I20260923 17:35:03.301360 25540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "8f5c713d883443928935e351ba8b0be2"
format_stamp: "Formatted at 2026-09-23 17:35:03 on dist-test-slave-ppjl"
server_key: "64f9057dad3ac3dc4be15066585a84f6"
server_key_iv: "dd008d3362309bf0f5e089e45b47e2da"
server_key_version: "encryptionkey@0"
I20260923 17:35:03.307143 25540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:35:03.311466 25554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:03.313024 25540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:35:03.313222 25540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "8f5c713d883443928935e351ba8b0be2"
format_stamp: "Formatted at 2026-09-23 17:35:03 on dist-test-slave-ppjl"
server_key: "64f9057dad3ac3dc4be15066585a84f6"
server_key_iv: "dd008d3362309bf0f5e089e45b47e2da"
server_key_version: "encryptionkey@0"
I20260923 17:35:03.313462 25540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:03.331634 25540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:03.336046 25540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:03.336335 25540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:03.354053 25540 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:33339
I20260923 17:35:03.354060 25605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:33339 every 8 connection(s)
I20260923 17:35:03.354988 25540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:35:03.357314 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 25540
I20260923 17:35:03.357553 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:35:03.358335 21441 external_mini_cluster.cc:1428] Setting key 4ed32f578710e9f661cb7a4c7270aedc
I20260923 17:35:03.359516 25606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:03.372596 25606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2: Bootstrap starting.
Sep 23 17:35:03 dist-test-slave-ppjl krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:35:03.376613 25606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:03.377774 25606 log.cc:824] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:03.380774 25606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2: No bootstrap required, opened a new log
I20260923 17:35:03.381804 25609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:03.360716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.20.240.126:33339)
0923 17:35:03.361599 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:03.361612 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:03.361656 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:35:03.363339 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:03.363365 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:03.363833 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:03.364282 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:03.365866 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:03.367368 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:03.368727 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:03.369496 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:03.369674 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:03.374069 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:03.374078 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:03.374096 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:03.374139 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:03.377374 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:03.378264 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:03.378271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:03.378278 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:03.378479 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:03.378915 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:03.378919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:03.378921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:03.379536 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:03.379845 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:03.380309 (+ 464us) server_negotiation.cc:299] Negotiation successful
0923 17:35:03.380751 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":417,"threads_started":1}
I20260923 17:35:03.388768 25606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5c713d883443928935e351ba8b0be2" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33339 } }
I20260923 17:35:03.389139 25606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:03.389240 25606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f5c713d883443928935e351ba8b0be2, State: Initialized, Role: FOLLOWER
I20260923 17:35:03.389816 25606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5c713d883443928935e351ba8b0be2" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33339 } }
I20260923 17:35:03.390031 25606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:35:03.390162 25606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:35:03.390339 25606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:03.392632 25606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5c713d883443928935e351ba8b0be2" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33339 } }
I20260923 17:35:03.393116 25606 leader_election.cc:302] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f5c713d883443928935e351ba8b0be2; no voters:
I20260923 17:35:03.393714 25606 leader_election.cc:288] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:35:03.393848 25611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:03.394477 25611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [term 1 LEADER]: Becoming Leader. State: Replica: 8f5c713d883443928935e351ba8b0be2, State: Running, Role: LEADER
I20260923 17:35:03.395040 25611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5c713d883443928935e351ba8b0be2" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33339 } }
I20260923 17:35:03.395939 25606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:35:03.397902 25613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f5c713d883443928935e351ba8b0be2. Latest consensus state: current_term: 1 leader_uuid: "8f5c713d883443928935e351ba8b0be2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5c713d883443928935e351ba8b0be2" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33339 } } }
I20260923 17:35:03.397928 25612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f5c713d883443928935e351ba8b0be2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f5c713d883443928935e351ba8b0be2" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33339 } } }
I20260923 17:35:03.398340 25613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:03.398486 25612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:03.400219 25619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:35:03.405172 25619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:35:03.414987 25619 catalog_manager.cc:1347] Generated new cluster ID: f1fb20ba5395420090df9cecda37af6e
I20260923 17:35:03.415246 25619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:35:03.427848 25619 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:35:03.429453 25619 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:35:03.440527 25619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2: Generated new TSK 0
I20260923 17:35:03.441740 25619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:35:03.495683 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33339
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:03.820957 25632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:03.821274 25632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:03.821374 25632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:03.828835 25632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:03.828969 25632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:03.829087 25632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:35:03.839193 25632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:03.840704 25632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:03.842126 25632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:03.855470 25640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:03.855525 25638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:03.855491 25637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:03.856513 25632 server_base.cc:1034] running on GCE node
I20260923 17:35:03.857218 25632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:03.858230 25632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:03.859524 25632 hybrid_clock.cc:648] HybridClock initialized: now 1790184903859452 us; error 46 us; skew 500 ppm
I20260923 17:35:03.859899 25632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:03 dist-test-slave-ppjl krb5kdc[25527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184903, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:03.863234 25632 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:35:03.864380 25632 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:03.864496 25632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:03.864821 25632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:03.867961 25632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4"
format_stamp: "Formatted at 2026-09-23 17:35:03 on dist-test-slave-ppjl"
server_key: "39a125f2f1569245184da18254e4dcb1"
server_key_iv: "9b21c9d23631900ae2537148afd420f2"
server_key_version: "encryptionkey@0"
I20260923 17:35:03.868832 25632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4"
format_stamp: "Formatted at 2026-09-23 17:35:03 on dist-test-slave-ppjl"
server_key: "39a125f2f1569245184da18254e4dcb1"
server_key_iv: "9b21c9d23631900ae2537148afd420f2"
server_key_version: "encryptionkey@0"
I20260923 17:35:03.874636 25632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:35:03.878683 25646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:03.880254 25632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:35:03.880432 25632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4"
format_stamp: "Formatted at 2026-09-23 17:35:03 on dist-test-slave-ppjl"
server_key: "39a125f2f1569245184da18254e4dcb1"
server_key_iv: "9b21c9d23631900ae2537148afd420f2"
server_key_version: "encryptionkey@0"
I20260923 17:35:03.880654 25632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:03.893340 25632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:03.898443 25632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:03.898712 25632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:03.899761 25632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:03.901255 25632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:03.901348 25632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:03.901448 25632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:03.901513 25632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:03.932125 25632 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:42663
I20260923 17:35:03.932202 25758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:42663 every 8 connection(s)
I20260923 17:35:03.932961 25632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:35:03.936209 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 25632
I20260923 17:35:03.936430 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:35:03.937271 21441 external_mini_cluster.cc:1428] Setting key 138b0fd8db7cb86f32678ba87ecef69b
Sep 23 17:35:03 dist-test-slave-ppjl krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184903, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:35:03.956064 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:03.935507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:59809 (local address 127.20.240.126:33339)
0923 17:35:03.936095 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:03.936102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:03.937141 (+ 1039us) server_negotiation.cc:407] Connection header received
0923 17:35:03.939287 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:03.939292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:03.939372 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:03.939496 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:03.941286 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:03.942214 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:03.944056 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:03.944679 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:03.944775 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:03.948941 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:03.948948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:03.948952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:03.948998 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:03.952228 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:03.953314 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:03.953319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:03.953322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:03.953460 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:03.954034 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:03.954039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:03.954041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:03.954352 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:03.954484 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:03.955557 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0923 17:35:03.955763 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":266,"threads_started":1}
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
I20260923 17:35:03.957512 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:03.936100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:33339 (local address 127.20.240.65:59809)
0923 17:35:03.936931 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:03.936984 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:03.938968 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:03.939733 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:03.939743 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:03.940381 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:03.941100 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:03.941122 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:03.942375 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:03.942379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:03.943889 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:03.943903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:03.944923 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:03.944931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:03.945301 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:03.946010 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:03.946045 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:03.948677 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:03.952438 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:03.952449 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:03.952469 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:03.953123 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:03.953579 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:03.953585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:03.953589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:03.953858 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:03.954497 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:03.954508 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:03.955002 (+ 494us) client_negotiation.cc:769] Sending connection context
0923 17:35:03.955392 (+ 390us) client_negotiation.cc:240] Negotiation successful
0923 17:35:03.955990 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":303,"threads_started":1}
I20260923 17:35:03.959342 25759 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
I20260923 17:35:03.959697 25759 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:03.960415 25759 heartbeater.cc:511] Master 127.20.240.126:33339 requested a full tablet report, sending...
I20260923 17:35:03.962661 25571 ts_manager.cc:194] Registered new tserver with Master: ae15c40adb6749a6a9fe1c64ef93e6e4 (127.20.240.65:42663)
I20260923 17:35:03.964236 25571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:59809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:35:03.982965 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33339
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:04.307281 25765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:04.307612 25765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:04.307725 25765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:04.315120 25765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:04.315297 25765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:04.315459 25765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:35:04.325512 25765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:04.326845 25765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:04.328222 25765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:04.340883 25770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:04.340866 25771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:04.341178 25773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:04.341892 25765 server_base.cc:1034] running on GCE node
I20260923 17:35:04.342522 25765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:04.343513 25765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:04.344764 25765 hybrid_clock.cc:648] HybridClock initialized: now 1790184904344685 us; error 62 us; skew 500 ppm
I20260923 17:35:04.345109 25765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:04 dist-test-slave-ppjl krb5kdc[25527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184904, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:04.348280 25765 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:35:04.349449 25765 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:04.349553 25765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:04.349866 25765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:04.352852 25765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "21775c2701894b12ac223733c9589cc2"
format_stamp: "Formatted at 2026-09-23 17:35:04 on dist-test-slave-ppjl"
server_key: "7c3acbdd9a3d50dcbf72714eff5b941f"
server_key_iv: "63387067ee15f5305f60f24f01fdc8ae"
server_key_version: "encryptionkey@0"
I20260923 17:35:04.353703 25765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "21775c2701894b12ac223733c9589cc2"
format_stamp: "Formatted at 2026-09-23 17:35:04 on dist-test-slave-ppjl"
server_key: "7c3acbdd9a3d50dcbf72714eff5b941f"
server_key_iv: "63387067ee15f5305f60f24f01fdc8ae"
server_key_version: "encryptionkey@0"
I20260923 17:35:04.358968 25765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 17:35:04.363090 25779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:04.364733 25765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:35:04.364898 25765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "21775c2701894b12ac223733c9589cc2"
format_stamp: "Formatted at 2026-09-23 17:35:04 on dist-test-slave-ppjl"
server_key: "7c3acbdd9a3d50dcbf72714eff5b941f"
server_key_iv: "63387067ee15f5305f60f24f01fdc8ae"
server_key_version: "encryptionkey@0"
I20260923 17:35:04.365118 25765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:04.394345 25765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:04.399096 25765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:04.399394 25765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:04.400460 25765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:04.401937 25765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:04.402026 25765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:04.402122 25765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:04.402174 25765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:04.437733 25765 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:33687
I20260923 17:35:04.437772 25891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:33687 every 8 connection(s)
I20260923 17:35:04.438539 25765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:35:04.441684 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 25765
I20260923 17:35:04.441879 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:35:04.442302 21441 external_mini_cluster.cc:1428] Setting key 5610e1f7b0177af695585b64d571be35
Sep 23 17:35:04 dist-test-slave-ppjl krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184904, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:35:04.457792 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:04.440981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:45463 (local address 127.20.240.126:33339)
0923 17:35:04.441169 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:04.441176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:04.442401 (+ 1225us) server_negotiation.cc:407] Connection header received
0923 17:35:04.444408 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:04.444413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:04.444489 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:04.444569 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:04.445990 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:04.446652 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:04.448189 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:04.448597 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:04.448712 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:04.452309 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:04.452315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:04.452318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:04.452351 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:04.454865 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:04.455610 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:04.455615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:04.455617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:04.455737 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:04.456128 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:04.456131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:04.456133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:04.456403 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:04.456530 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:04.457228 (+ 698us) server_negotiation.cc:299] Negotiation successful
0923 17:35:04.457455 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
I20260923 17:35:04.458788 25894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:04.441479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:33339 (local address 127.20.240.66:45463)
0923 17:35:04.442234 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:04.442287 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:04.444158 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:04.444798 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:04.444807 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:04.445303 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:04.445834 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:04.445851 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:04.446798 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:04.446801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:04.448058 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:04.448070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:04.448805 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:04.448812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:04.449022 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:04.449707 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:04.449739 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:04.452082 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:04.454996 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:04.455005 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:04.455023 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:04.455492 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:04.455837 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:04.455842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:04.455846 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:04.456021 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:04.456519 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:04.456527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:04.456876 (+ 349us) client_negotiation.cc:769] Sending connection context
0923 17:35:04.457118 (+ 242us) client_negotiation.cc:240] Negotiation successful
0923 17:35:04.457648 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":302,"threads_started":1}
I20260923 17:35:04.460464 25892 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33339
I20260923 17:35:04.460834 25892 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
I20260923 17:35:04.461633 25892 heartbeater.cc:511] Master 127.20.240.126:33339 requested a full tablet report, sending...
I20260923 17:35:04.463332 25571 ts_manager.cc:194] Registered new tserver with Master: 21775c2701894b12ac223733c9589cc2 (127.20.240.66:33687)
I20260923 17:35:04.464377 25571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:45463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:35:04.486977 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33339
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:04.810990 25897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:04.811334 25897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:04.811442 25897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:04.818956 25897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:04.819077 25897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:04.819238 25897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:35:04.829233 25897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:04.830519 25897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:04.831796 25897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:04.844543 25905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:04.844718 25903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:04.845059 25902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:04.845549 25897 server_base.cc:1034] running on GCE node
I20260923 17:35:04.846103 25897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:04.847080 25897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:04.848305 25897 hybrid_clock.cc:648] HybridClock initialized: now 1790184904848257 us; error 38 us; skew 500 ppm
I20260923 17:35:04.848615 25897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:04 dist-test-slave-ppjl krb5kdc[25527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184904, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:04.851784 25897 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:35:04.853024 25897 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:04.853129 25897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:04.853425 25897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:04.856619 25897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "5c868949f27542e495239bf26fcac6bb"
format_stamp: "Formatted at 2026-09-23 17:35:04 on dist-test-slave-ppjl"
server_key: "1c2c188f47ff25086887839971e0135f"
server_key_iv: "ac2dca285db83b548b18b34d3237d6db"
server_key_version: "encryptionkey@0"
I20260923 17:35:04.857414 25897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "5c868949f27542e495239bf26fcac6bb"
format_stamp: "Formatted at 2026-09-23 17:35:04 on dist-test-slave-ppjl"
server_key: "1c2c188f47ff25086887839971e0135f"
server_key_iv: "ac2dca285db83b548b18b34d3237d6db"
server_key_version: "encryptionkey@0"
I20260923 17:35:04.862672 25897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:35:04.866241 25911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:04.867746 25897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260923 17:35:04.867914 25897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "5c868949f27542e495239bf26fcac6bb"
format_stamp: "Formatted at 2026-09-23 17:35:04 on dist-test-slave-ppjl"
server_key: "1c2c188f47ff25086887839971e0135f"
server_key_iv: "ac2dca285db83b548b18b34d3237d6db"
server_key_version: "encryptionkey@0"
I20260923 17:35:04.868130 25897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:04.887923 25897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:04.891939 25897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:04.892197 25897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:04.893198 25897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:04.894640 25897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:04.894737 25897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:04.894851 25897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:04.894928 25897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:04.926915 25897 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:42131
I20260923 17:35:04.926949 26023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:42131 every 8 connection(s)
I20260923 17:35:04.927820 25897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:35:04.936031 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 25897
I20260923 17:35:04.936231 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:35:04.936637 21441 external_mini_cluster.cc:1428] Setting key 360632a56dd50f2242ada9b35bca3975
Sep 23 17:35:04 dist-test-slave-ppjl krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184904, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:35:04.949349 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:04.930379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:53167 (local address 127.20.240.126:33339)
0923 17:35:04.930560 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:04.930564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:04.931850 (+ 1286us) server_negotiation.cc:407] Connection header received
0923 17:35:04.934346 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:04.934351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:04.934502 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:04.934618 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:04.936797 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:04.937760 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:04.939403 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:04.939839 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:04.939977 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:04.943586 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:04.943592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:04.943596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:04.943634 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:04.946171 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:04.946973 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:04.946978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:04.946980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:04.947129 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:04.947605 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:04.947609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:04.947611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:04.947916 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:04.948037 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:04.948795 (+ 758us) server_negotiation.cc:299] Negotiation successful
0923 17:35:04.948982 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:35:04.950178 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:04.930829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:33339 (local address 127.20.240.67:53167)
0923 17:35:04.931671 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:04.931718 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:04.933992 (+ 2274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:04.934946 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:04.934958 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:04.935779 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:04.936565 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:04.936603 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:04.937922 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:04.937928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:04.939257 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:04.939271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:04.940069 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:04.940076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:04.940326 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:04.941034 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:04.941059 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:04.943309 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:04.946340 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:04.946349 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:04.946365 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:04.946828 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:04.947272 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:04.947275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:04.947278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:04.947480 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:04.948020 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:04.948027 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:04.948404 (+ 377us) client_negotiation.cc:769] Sending connection context
0923 17:35:04.948635 (+ 231us) client_negotiation.cc:240] Negotiation successful
0923 17:35:04.949067 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":301,"threads_started":1}
I20260923 17:35:04.951743 26024 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33339
I20260923 17:35:04.952049 26024 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:04.952775 26024 heartbeater.cc:511] Master 127.20.240.126:33339 requested a full tablet report, sending...
I20260923 17:35:04.954082 25570 ts_manager.cc:194] Registered new tserver with Master: 5c868949f27542e495239bf26fcac6bb (127.20.240.67:42131)
I20260923 17:35:04.954982 25570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:53167
I20260923 17:35:04.964993 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:35:04.967945 25759 heartbeater.cc:503] Master 127.20.240.126:33339 was elected leader, sending a full tablet report...
I20260923 17:35:04.982576 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:04.969670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.20.240.126:33339)
0923 17:35:04.969827 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:04.969831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:04.970410 (+ 579us) server_negotiation.cc:407] Connection header received
0923 17:35:04.970593 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:04.970599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:04.970666 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:04.970741 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:04.972308 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:04.973142 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:04.974417 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:04.974851 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:04.974961 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:04.977215 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:04.977221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:04.977224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:04.977261 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:04.979784 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:04.980493 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:04.980498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:04.980502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:04.980645 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:04.981021 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:04.981024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:04.981027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:04.981377 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:04.981620 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:04.982017 (+ 397us) server_negotiation.cc:299] Negotiation successful
0923 17:35:04.982251 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 17:35:04.987197 25570 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:35:04.994035 25570 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:35:05.028365 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:05.007769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.20.240.66:33687)
0923 17:35:05.018752 (+ 10983us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:05.018762 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:05.019987 (+ 1225us) server_negotiation.cc:407] Connection header received
0923 17:35:05.020191 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:05.020199 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:05.020434 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:05.020646 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:05.021495 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.023077 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.025772 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.027028 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.027200 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:05.027415 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:05.027822 (+ 407us) server_negotiation.cc:299] Negotiation successful
0923 17:35:05.028016 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10754,"thread_start_us":10992,"threads_started":1}
I20260923 17:35:05.028436 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:05.006714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:42663 (local address 127.0.0.1:45272)
0923 17:35:05.013897 (+ 7183us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:05.013947 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:05.014173 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:05.021392 (+ 7219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:05.021397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:05.021421 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:05.021814 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:05.021827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.023635 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.023639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:05.025798 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:05.025814 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.027741 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.027748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:05.027898 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:05.027927 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 17:35:05.028001 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 17:35:05.028123 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6874,"thread_start_us":8137,"threads_started":1}
I20260923 17:35:05.028813 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:05.007307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:33687 (local address 127.0.0.1:43390)
0923 17:35:05.019869 (+ 12562us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:05.019889 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:05.020031 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:05.020749 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:05.020757 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:05.020792 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:05.021294 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:05.021309 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.023242 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.023248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:05.025576 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:05.025590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.027359 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.027366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:05.027549 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:05.027601 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 17:35:05.027754 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 17:35:05.027959 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12407,"mutex_wait_us":21,"thread_start_us":13369,"threads_started":1}
I20260923 17:35:05.029264 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:05.007352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59026 (local address 127.20.240.67:42131)
0923 17:35:05.020604 (+ 13252us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:05.020613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:05.021299 (+ 686us) server_negotiation.cc:407] Connection header received
0923 17:35:05.021424 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:05.021432 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:05.021648 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:05.021861 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:05.022717 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.024100 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.026659 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.027919 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.028039 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:05.028238 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:05.028807 (+ 569us) server_negotiation.cc:299] Negotiation successful
0923 17:35:05.028970 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13069,"thread_start_us":12925,"threads_started":1}
I20260923 17:35:05.029428 26037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:05.007030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.20.240.65:42663)
0923 17:35:05.019502 (+ 12472us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:05.019511 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:05.019543 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:35:05.019706 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:05.019716 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:05.020955 (+ 1239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:05.021167 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:05.022003 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.023469 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.025983 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.027226 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.027358 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:05.027594 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:05.028949 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0923 17:35:05.029120 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12307,"thread_start_us":12157,"threads_started":1}
I20260923 17:35:05.030205 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:05.007817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:42131 (local address 127.0.0.1:59026)
0923 17:35:05.021023 (+ 13206us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:05.021041 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:05.021166 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:05.022131 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:05.022135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:05.022160 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:05.022522 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:05.022532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.024257 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.024263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:05.026486 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:05.026501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.028428 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.028435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:05.028609 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:05.028639 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 17:35:05.028724 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 17:35:05.028857 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13065,"mutex_wait_us":2,"thread_start_us":11722,"threads_started":1}
I20260923 17:35:05.032718 25694 tablet_service.cc:1467] Processing CreateTablet for tablet 75295b30a0204982b856363386ee96ee (DEFAULT_TABLE table=test-table [id=271e35f4138542c2a70aa8f346471317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:05.032801 25827 tablet_service.cc:1467] Processing CreateTablet for tablet 75295b30a0204982b856363386ee96ee (DEFAULT_TABLE table=test-table [id=271e35f4138542c2a70aa8f346471317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:05.033963 25694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75295b30a0204982b856363386ee96ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:05.033538 25959 tablet_service.cc:1467] Processing CreateTablet for tablet 75295b30a0204982b856363386ee96ee (DEFAULT_TABLE table=test-table [id=271e35f4138542c2a70aa8f346471317]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:05.034124 25827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75295b30a0204982b856363386ee96ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:05.034785 25959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75295b30a0204982b856363386ee96ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:05.046092 26042 tablet_bootstrap.cc:492] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4: Bootstrap starting.
I20260923 17:35:05.048931 26043 tablet_bootstrap.cc:492] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2: Bootstrap starting.
I20260923 17:35:05.051555 26042 tablet_bootstrap.cc:654] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:05.052309 26044 tablet_bootstrap.cc:492] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb: Bootstrap starting.
I20260923 17:35:05.052628 26043 tablet_bootstrap.cc:654] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:05.053265 26042 log.cc:824] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:05.053999 26043 log.cc:824] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:05.055588 26044 tablet_bootstrap.cc:654] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:05.056639 26042 tablet_bootstrap.cc:492] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4: No bootstrap required, opened a new log
I20260923 17:35:05.056833 26044 log.cc:824] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:05.056905 26042 ts_tablet_manager.cc:1397] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260923 17:35:05.057183 26043 tablet_bootstrap.cc:492] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2: No bootstrap required, opened a new log
I20260923 17:35:05.057474 26043 ts_tablet_manager.cc:1397] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260923 17:35:05.060045 26044 tablet_bootstrap.cc:492] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb: No bootstrap required, opened a new log
I20260923 17:35:05.060341 26044 ts_tablet_manager.cc:1397] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260923 17:35:05.064878 26042 raft_consensus.cc:348] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 42663 } } peers { permanent_uuid: "5c868949f27542e495239bf26fcac6bb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42131 } } peers { permanent_uuid: "21775c2701894b12ac223733c9589cc2" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 33687 } }
I20260923 17:35:05.065272 26042 raft_consensus.cc:374] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:05.065385 26042 raft_consensus.cc:729] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae15c40adb6749a6a9fe1c64ef93e6e4, State: Initialized, Role: FOLLOWER
I20260923 17:35:05.066012 26042 consensus_queue.cc:260] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 42663 } } peers { permanent_uuid: "5c868949f27542e495239bf26fcac6bb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42131 } } peers { permanent_uuid: "21775c2701894b12ac223733c9589cc2" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 33687 } }
I20260923 17:35:05.067514 26043 raft_consensus.cc:348] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 42663 } } peers { permanent_uuid: "5c868949f27542e495239bf26fcac6bb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42131 } } peers { permanent_uuid: "21775c2701894b12ac223733c9589cc2" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 33687 } }
I20260923 17:35:05.067940 26042 ts_tablet_manager.cc:1428] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260923 17:35:05.067960 26043 raft_consensus.cc:374] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:05.068046 26043 raft_consensus.cc:729] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21775c2701894b12ac223733c9589cc2, State: Initialized, Role: FOLLOWER
I20260923 17:35:05.068624 26043 consensus_queue.cc:260] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 42663 } } peers { permanent_uuid: "5c868949f27542e495239bf26fcac6bb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42131 } } peers { permanent_uuid: "21775c2701894b12ac223733c9589cc2" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 33687 } }
I20260923 17:35:05.070909 26044 raft_consensus.cc:348] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 42663 } } peers { permanent_uuid: "5c868949f27542e495239bf26fcac6bb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42131 } } peers { permanent_uuid: "21775c2701894b12ac223733c9589cc2" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 33687 } }
I20260923 17:35:05.071290 25892 heartbeater.cc:503] Master 127.20.240.126:33339 was elected leader, sending a full tablet report...
I20260923 17:35:05.071381 26044 raft_consensus.cc:374] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:05.071470 26043 ts_tablet_manager.cc:1428] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260923 17:35:05.071486 26044 raft_consensus.cc:729] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c868949f27542e495239bf26fcac6bb, State: Initialized, Role: FOLLOWER
I20260923 17:35:05.072122 26044 consensus_queue.cc:260] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 42663 } } peers { permanent_uuid: "5c868949f27542e495239bf26fcac6bb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42131 } } peers { permanent_uuid: "21775c2701894b12ac223733c9589cc2" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 33687 } }
I20260923 17:35:05.073220 26024 heartbeater.cc:503] Master 127.20.240.126:33339 was elected leader, sending a full tablet report...
I20260923 17:35:05.073982 26044 ts_tablet_manager.cc:1428] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260923 17:35:05.179327 26025 tablet.cc:2369] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:35:05.185271 25760 tablet.cc:2369] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:35:05.190459 25893 tablet.cc:2369] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:35:05.289408 26049 raft_consensus.cc:482] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:35:05.289702 26049 raft_consensus.cc:504] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 42663 } } peers { permanent_uuid: "5c868949f27542e495239bf26fcac6bb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42131 } } peers { permanent_uuid: "21775c2701894b12ac223733c9589cc2" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 33687 } }
I20260923 17:35:05.291548 26049 leader_election.cc:288] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae15c40adb6749a6a9fe1c64ef93e6e4 (127.20.240.65:42663), 5c868949f27542e495239bf26fcac6bb (127.20.240.67:42131)
I20260923 17:35:05.298514 26037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:05.292162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:48717 (local address 127.20.240.65:42663)
0923 17:35:05.292399 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:05.292406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:05.293043 (+ 637us) server_negotiation.cc:407] Connection header received
0923 17:35:05.293258 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:05.293262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:05.293334 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:05.293409 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:05.293952 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.294698 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.296890 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.297639 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.297790 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:05.297863 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:05.298148 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 17:35:05.298270 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 17:35:05.298507 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:05.292258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:50133 (local address 127.20.240.67:42131)
0923 17:35:05.292487 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:05.292493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:05.293219 (+ 726us) server_negotiation.cc:407] Connection header received
0923 17:35:05.293476 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:05.293481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:05.293544 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:05.293626 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:05.294180 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.294876 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.296819 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.297594 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.297768 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:05.297847 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:05.298131 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 17:35:05.298256 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 17:35:05.298664 26052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:05.291875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:42663 (local address 127.20.240.66:48717)
0923 17:35:05.292912 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:05.292937 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:05.293096 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:05.293444 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:05.293448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:05.293473 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:05.293802 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:05.293816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.294836 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.294840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:05.296667 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:05.296682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.297858 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.297864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:05.297998 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:05.298028 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 17:35:05.298106 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 17:35:05.298232 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":51,"thread_start_us":897,"threads_started":1}
I20260923 17:35:05.299037 26051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:05.292045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:42131 (local address 127.20.240.66:50133)
0923 17:35:05.293057 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:05.293082 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:05.293317 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:05.293663 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:05.293667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:05.293695 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:05.294031 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:05.294040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.295013 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.295018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:05.296665 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:05.296681 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.297824 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.297832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:05.297986 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:05.298012 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:35:05.298092 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 17:35:05.298232 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":508,"threads_started":1}
I20260923 17:35:05.299726 25714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75295b30a0204982b856363386ee96ee" candidate_uuid: "21775c2701894b12ac223733c9589cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4" is_pre_election: true
I20260923 17:35:05.299959 25979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75295b30a0204982b856363386ee96ee" candidate_uuid: "21775c2701894b12ac223733c9589cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c868949f27542e495239bf26fcac6bb" is_pre_election: true
I20260923 17:35:05.300257 25714 raft_consensus.cc:2454] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21775c2701894b12ac223733c9589cc2 in term 0.
I20260923 17:35:05.300450 25979 raft_consensus.cc:2454] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21775c2701894b12ac223733c9589cc2 in term 0.
I20260923 17:35:05.300920 25780 leader_election.cc:302] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21775c2701894b12ac223733c9589cc2, ae15c40adb6749a6a9fe1c64ef93e6e4; no voters:
I20260923 17:35:05.301355 26049 raft_consensus.cc:2781] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:35:05.301479 26049 raft_consensus.cc:482] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:35:05.301584 26049 raft_consensus.cc:3037] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:05.303812 26049 raft_consensus.cc:504] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 42663 } } peers { permanent_uuid: "5c868949f27542e495239bf26fcac6bb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42131 } } peers { permanent_uuid: "21775c2701894b12ac223733c9589cc2" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 33687 } }
I20260923 17:35:05.304438 26049 leader_election.cc:288] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [CANDIDATE]: Term 1 election: Requested vote from peers ae15c40adb6749a6a9fe1c64ef93e6e4 (127.20.240.65:42663), 5c868949f27542e495239bf26fcac6bb (127.20.240.67:42131)
I20260923 17:35:05.304955 25979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75295b30a0204982b856363386ee96ee" candidate_uuid: "21775c2701894b12ac223733c9589cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c868949f27542e495239bf26fcac6bb"
I20260923 17:35:05.304965 25714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75295b30a0204982b856363386ee96ee" candidate_uuid: "21775c2701894b12ac223733c9589cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4"
I20260923 17:35:05.305145 25979 raft_consensus.cc:3037] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:05.305142 25714 raft_consensus.cc:3037] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:05.307296 25979 raft_consensus.cc:2454] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21775c2701894b12ac223733c9589cc2 in term 1.
I20260923 17:35:05.307307 25714 raft_consensus.cc:2454] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21775c2701894b12ac223733c9589cc2 in term 1.
I20260923 17:35:05.307734 25781 leader_election.cc:302] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21775c2701894b12ac223733c9589cc2, 5c868949f27542e495239bf26fcac6bb, ae15c40adb6749a6a9fe1c64ef93e6e4; no voters:
I20260923 17:35:05.308044 26049 raft_consensus.cc:2781] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:05.308513 26049 raft_consensus.cc:686] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [term 1 LEADER]: Becoming Leader. State: Replica: 21775c2701894b12ac223733c9589cc2, State: Running, Role: LEADER
I20260923 17:35:05.308995 26049 consensus_queue.cc:237] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 42663 } } peers { permanent_uuid: "5c868949f27542e495239bf26fcac6bb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42131 } } peers { permanent_uuid: "21775c2701894b12ac223733c9589cc2" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 33687 } }
I20260923 17:35:05.313894 25570 catalog_manager.cc:5495] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21775c2701894b12ac223733c9589cc2 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "21775c2701894b12ac223733c9589cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 42663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c868949f27542e495239bf26fcac6bb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21775c2701894b12ac223733c9589cc2" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 33687 } health_report { overall_health: HEALTHY } } }
I20260923 17:35:05.398548 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:05.393460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43394 (local address 127.20.240.66:33687)
0923 17:35:05.393685 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:05.393689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:05.393711 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:35:05.393810 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:05.393814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:05.393901 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:05.394043 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:35:05.394415 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.395145 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.396541 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.396990 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.397087 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:05.397557 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:35:05.398100 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:35:05.398225 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:05.398262 (+ 37us) server_negotiation.cc:299] Negotiation successful
0923 17:35:05.398330 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:35:05.406641 25979 raft_consensus.cc:1261] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb [term 1 FOLLOWER]: Refusing update from remote peer 21775c2701894b12ac223733c9589cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:05.406663 25714 raft_consensus.cc:1261] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4 [term 1 FOLLOWER]: Refusing update from remote peer 21775c2701894b12ac223733c9589cc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:05.407605 26053 consensus_queue.cc:1035] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae15c40adb6749a6a9fe1c64ef93e6e4" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 42663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:05.407948 26049 consensus_queue.cc:1035] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c868949f27542e495239bf26fcac6bb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:05.419718 26056 mvcc.cc:205] Tried to move back new op lower bound from 7332597372536508416 to 7332597372149157888. Current Snapshot: MvccSnapshot[applied={T|T < 7332597372536508416}]
I20260923 17:35:05.421921 26057 mvcc.cc:205] Tried to move back new op lower bound from 7332597372536508416 to 7332597372149157888. Current Snapshot: MvccSnapshot[applied={T|T < 7332597372536508416}]
I20260923 17:35:05.436811 25571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:05.437080 25571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:05.441558 25571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f5c713d883443928935e351ba8b0be2: Sending DeleteTablet for 3 replicas of tablet 75295b30a0204982b856363386ee96ee
I20260923 17:35:05.442945 25694 tablet_service.cc:1514] Processing DeleteTablet for tablet 75295b30a0204982b856363386ee96ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:05 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:45272
I20260923 17:35:05.442943 25959 tablet_service.cc:1514] Processing DeleteTablet for tablet 75295b30a0204982b856363386ee96ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:05 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:59026
I20260923 17:35:05.446632 25827 tablet_service.cc:1514] Processing DeleteTablet for tablet 75295b30a0204982b856363386ee96ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:05 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:43390
I20260923 17:35:05.448798 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 25632
I20260923 17:35:05.454954 26068 tablet_replica.cc:330] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb: stopping tablet replica
I20260923 17:35:05.455636 26068 raft_consensus.cc:2229] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:05.456272 26068 raft_consensus.cc:2258] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:05.457077 26067 tablet_replica.cc:330] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4: stopping tablet replica
I20260923 17:35:05.457530 26067 raft_consensus.cc:2229] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:05.458056 26067 raft_consensus.cc:2258] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:05.459185 26068 ts_tablet_manager.cc:1905] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:05.460318 26067 ts_tablet_manager.cc:1905] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:05.467401 26067 ts_tablet_manager.cc:1918] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:05.467706 26067 log.cc:1196] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/75295b30a0204982b856363386ee96ee
I20260923 17:35:05.467828 26069 tablet_replica.cc:330] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2: stopping tablet replica
I20260923 17:35:05.468325 26069 raft_consensus.cc:2229] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:35:05.468542 26067 ts_tablet_manager.cc:1939] T 75295b30a0204982b856363386ee96ee P ae15c40adb6749a6a9fe1c64ef93e6e4: Deleting consensus metadata
I20260923 17:35:05.469153 26069 raft_consensus.cc:2258] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:05.470702 25555 catalog_manager.cc:4841] TS ae15c40adb6749a6a9fe1c64ef93e6e4 (127.20.240.65:42663): tablet 75295b30a0204982b856363386ee96ee (table test-table [id=271e35f4138542c2a70aa8f346471317]) successfully deleted
I20260923 17:35:05.471381 26069 ts_tablet_manager.cc:1905] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:05.474128 26068 ts_tablet_manager.cc:1918] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:05.474388 26068 log.cc:1196] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/75295b30a0204982b856363386ee96ee
I20260923 17:35:05.475030 26068 ts_tablet_manager.cc:1939] T 75295b30a0204982b856363386ee96ee P 5c868949f27542e495239bf26fcac6bb: Deleting consensus metadata
I20260923 17:35:05.477365 25556 catalog_manager.cc:4841] TS 5c868949f27542e495239bf26fcac6bb (127.20.240.67:42131): tablet 75295b30a0204982b856363386ee96ee (table test-table [id=271e35f4138542c2a70aa8f346471317]) successfully deleted
I20260923 17:35:05.478777 26069 ts_tablet_manager.cc:1918] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:05.478950 26069 log.cc:1196] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/75295b30a0204982b856363386ee96ee
I20260923 17:35:05.479552 26069 ts_tablet_manager.cc:1939] T 75295b30a0204982b856363386ee96ee P 21775c2701894b12ac223733c9589cc2: Deleting consensus metadata
Sep 23 17:35:05 dist-test-slave-ppjl krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:35:05.481133 25557 catalog_manager.cc:4841] TS 21775c2701894b12ac223733c9589cc2 (127.20.240.66:33687): tablet 75295b30a0204982b856363386ee96ee (table test-table [id=271e35f4138542c2a70aa8f346471317]) successfully deleted
I20260923 17:35:05.486658 26037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:05.450181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.20.240.65:42663)
0923 17:35:05.450358 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:05.450364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:05.469038 (+ 18674us) server_negotiation.cc:407] Connection header received
0923 17:35:05.469970 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:05.469977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:05.470066 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:05.470143 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:05.472123 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.473205 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.475037 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.475630 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.475740 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:05.480849 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:05.480867 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:05.480887 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:05.480934 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:05.483839 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:05.484673 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:05.484681 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:05.484683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:05.484834 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:05.485233 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:05.485236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:05.485239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:05.485663 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:05.485907 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:05.486202 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 17:35:05.486386 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:35:05.487393 25754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:05.580179 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 25632
I20260923 17:35:05.594049 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 25765
Sep 23 17:35:05 dist-test-slave-ppjl krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:35:05.610678 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:05.594802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.20.240.66:33687)
0923 17:35:05.594956 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:05.594960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:05.594977 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:35:05.595194 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:05.595199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:05.595276 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:05.595356 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:05.597014 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.597866 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.599687 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.600145 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.600249 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:05.605010 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:05.605028 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:05.605045 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:05.605086 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:05.607996 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:05.608648 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:05.608655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:05.608658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:05.608810 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:05.609162 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:05.609166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:05.609168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:05.609525 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:05.609853 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:05.610110 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 17:35:05.610319 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:35:05.611418 25887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:05.726838 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 25765
I20260923 17:35:05.742012 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 25897
Sep 23 17:35:05 dist-test-slave-ppjl krb5kdc[25527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:35:05.756618 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:05.742747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.20.240.67:42131)
0923 17:35:05.742934 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:05.742939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:05.742958 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:35:05.743091 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:05.743096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:05.743191 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:05.743275 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:05.744495 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.745192 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.746416 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:05.746825 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:05.746911 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:05.750915 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:05.750930 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:05.750947 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:05.750985 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:05.753660 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:05.754287 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:05.754294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:05.754296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:05.754427 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:05.754891 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:05.754894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:05.754896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:05.755548 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:05.755791 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:05.756145 (+ 354us) server_negotiation.cc:299] Negotiation successful
0923 17:35:05.756329 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:35:05.757351 26019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:05.845708 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 25897
I20260923 17:35:05.858767 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 25540
I20260923 17:35:05.860082 25601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:05.968992 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 25540
2026-09-23T17:35:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4242 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T17:35:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:35:06Z Disabled control of system clock
I20260923 17:35:06.031072 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:42495
--webserver_interface=127.20.240.126
--webserver_port=0
--builtin_ntp_servers=127.20.240.84:46735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:42495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 17:35:06.354920 26078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:06.355294 26078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:06.355430 26078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:06.363019 26078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:35:06.363137 26078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:06.363207 26078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:06.363288 26078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:35:06.363345 26078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:35:06.373072 26078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:46735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:42495
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:42495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:06.374502 26078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:06.376435 26078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:06.388423 26083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:06.388394 26086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:06.388394 26084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:06.389503 26078 server_base.cc:1034] running on GCE node
I20260923 17:35:06.390256 26078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:06.391714 26078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:06.392962 26078 hybrid_clock.cc:648] HybridClock initialized: now 1790184906392909 us; error 36 us; skew 500 ppm
I20260923 17:35:06.393383 26078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:35:06.395910 26078 webserver.cc:460] Webserver started at http://127.20.240.126:35509/ using document root <none> and password file <none>
I20260923 17:35:06.396608 26078 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:06.396699 26078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:06.397008 26078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:06.399969 26078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "460e973086164bf7ab83b5401bd9817c"
format_stamp: "Formatted at 2026-09-23 17:35:06 on dist-test-slave-ppjl"
server_key: "6c21f0d1054b51cc31e9813e6196a5ab"
server_key_iv: "ac9cba3f8cdda98dbd3147322b213558"
server_key_version: "encryptionkey@0"
I20260923 17:35:06.400772 26078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "460e973086164bf7ab83b5401bd9817c"
format_stamp: "Formatted at 2026-09-23 17:35:06 on dist-test-slave-ppjl"
server_key: "6c21f0d1054b51cc31e9813e6196a5ab"
server_key_iv: "ac9cba3f8cdda98dbd3147322b213558"
server_key_version: "encryptionkey@0"
I20260923 17:35:06.406128 26078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:35:06.409966 26092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:06.411661 26078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:35:06.411855 26078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "460e973086164bf7ab83b5401bd9817c"
format_stamp: "Formatted at 2026-09-23 17:35:06 on dist-test-slave-ppjl"
server_key: "6c21f0d1054b51cc31e9813e6196a5ab"
server_key_iv: "ac9cba3f8cdda98dbd3147322b213558"
server_key_version: "encryptionkey@0"
I20260923 17:35:06.412108 26078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:06.418186 26078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:06.419046 26078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:06.419374 26078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:06.436528 26143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:42495 every 8 connection(s)
I20260923 17:35:06.436522 26078 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:42495
I20260923 17:35:06.438011 26078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:35:06.439508 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 26078
I20260923 17:35:06.439703 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:35:06.440521 21441 external_mini_cluster.cc:1428] Setting key 460bdafb2f617be61bc3ab144bbc8f81
I20260923 17:35:06.443137 26144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:06.450611 26147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:06.442821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.20.240.126:42495)
0923 17:35:06.443896 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:06.443911 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:06.443963 (+ 52us) server_negotiation.cc:407] Connection header received
0923 17:35:06.446079 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:06.446111 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:06.446742 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:06.447304 (+ 562us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:06.448395 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:06.448404 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:06.448427 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:06.448457 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:06.448551 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:06.448574 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:06.448823 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:06.448959 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 17:35:06.449272 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":345,"threads_started":1}
I20260923 17:35:06.456833 26144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c: Bootstrap starting.
I20260923 17:35:06.459977 26144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:06.461014 26144 log.cc:824] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:06.463912 26144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c: No bootstrap required, opened a new log
I20260923 17:35:06.471235 26144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460e973086164bf7ab83b5401bd9817c" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42495 } }
I20260923 17:35:06.471640 26144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:06.471738 26144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 460e973086164bf7ab83b5401bd9817c, State: Initialized, Role: FOLLOWER
I20260923 17:35:06.472335 26144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460e973086164bf7ab83b5401bd9817c" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42495 } }
I20260923 17:35:06.472560 26144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:35:06.472692 26144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:35:06.472857 26144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:06.475301 26144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460e973086164bf7ab83b5401bd9817c" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42495 } }
I20260923 17:35:06.475837 26144 leader_election.cc:302] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 460e973086164bf7ab83b5401bd9817c; no voters:
I20260923 17:35:06.476411 26144 leader_election.cc:288] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:35:06.476537 26149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:06.477192 26149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [term 1 LEADER]: Becoming Leader. State: Replica: 460e973086164bf7ab83b5401bd9817c, State: Running, Role: LEADER
I20260923 17:35:06.477732 26149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460e973086164bf7ab83b5401bd9817c" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42495 } }
I20260923 17:35:06.478408 26144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:35:06.480235 26150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "460e973086164bf7ab83b5401bd9817c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460e973086164bf7ab83b5401bd9817c" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42495 } } }
I20260923 17:35:06.480530 26150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:06.480988 26151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 460e973086164bf7ab83b5401bd9817c. Latest consensus state: current_term: 1 leader_uuid: "460e973086164bf7ab83b5401bd9817c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "460e973086164bf7ab83b5401bd9817c" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42495 } } }
I20260923 17:35:06.481381 26151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:06.481554 26155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:35:06.487257 26155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:35:06.497336 26155 catalog_manager.cc:1347] Generated new cluster ID: a966cd6d903946ccbf1da6f35d0171dd
I20260923 17:35:06.497468 26155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:35:06.524690 26155 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:35:06.525182 26155 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:35:06.534197 26155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c: Generated new TSK 0
I20260923 17:35:06.535399 26155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:35:06.553658 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42495
--builtin_ntp_servers=127.20.240.84:46735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 17:35:06.882896 26168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:06.883337 26168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:06.883476 26168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:06.891556 26168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:06.891666 26168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:06.891800 26168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:35:06.901266 26168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:46735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:06.902619 26168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:06.904796 26168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:06.918362 26173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:06.918323 26174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:06.918385 26176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:06.919095 26168 server_base.cc:1034] running on GCE node
I20260923 17:35:06.919708 26168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:06.920648 26168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:06.921860 26168 hybrid_clock.cc:648] HybridClock initialized: now 1790184906921822 us; error 38 us; skew 500 ppm
I20260923 17:35:06.922207 26168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:35:06.924482 26168 webserver.cc:460] Webserver started at http://127.20.240.65:34017/ using document root <none> and password file <none>
I20260923 17:35:06.925144 26168 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:06.925230 26168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:06.925496 26168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:06.928428 26168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "b690fe42b7f140a4b36f6d0c04a4ae54"
format_stamp: "Formatted at 2026-09-23 17:35:06 on dist-test-slave-ppjl"
server_key: "1bd4fda2004967c22a4f2488f4a58ff8"
server_key_iv: "865c5d3af8cfe536b10b6a8f68369925"
server_key_version: "encryptionkey@0"
I20260923 17:35:06.929157 26168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "b690fe42b7f140a4b36f6d0c04a4ae54"
format_stamp: "Formatted at 2026-09-23 17:35:06 on dist-test-slave-ppjl"
server_key: "1bd4fda2004967c22a4f2488f4a58ff8"
server_key_iv: "865c5d3af8cfe536b10b6a8f68369925"
server_key_version: "encryptionkey@0"
I20260923 17:35:06.934387 26168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:35:06.938380 26182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:06.940011 26168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:35:06.940186 26168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "b690fe42b7f140a4b36f6d0c04a4ae54"
format_stamp: "Formatted at 2026-09-23 17:35:06 on dist-test-slave-ppjl"
server_key: "1bd4fda2004967c22a4f2488f4a58ff8"
server_key_iv: "865c5d3af8cfe536b10b6a8f68369925"
server_key_version: "encryptionkey@0"
I20260923 17:35:06.940400 26168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:06.946252 26168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:06.947082 26168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:06.947372 26168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:06.948288 26168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:06.949841 26168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:06.949929 26168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:06.950013 26168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:06.950066 26168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:06.980831 26168 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:36403
I20260923 17:35:06.980890 26294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:36403 every 8 connection(s)
I20260923 17:35:06.982096 26168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:35:06.984472 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 26168
I20260923 17:35:06.984649 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:35:06.985376 21441 external_mini_cluster.cc:1428] Setting key 31fed7882a634de800650ea2de8fa5d2
I20260923 17:35:06.988385 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42495
--builtin_ntp_servers=127.20.240.84:46735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 17:35:06.991310 26297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:06.984614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:57093 (local address 127.20.240.126:42495)
0923 17:35:06.985071 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:06.985078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:06.986493 (+ 1415us) server_negotiation.cc:407] Connection header received
0923 17:35:06.988750 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:06.988755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:06.988811 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:06.988922 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:06.990419 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:06.990423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:06.990427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:06.990447 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:06.990502 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:06.990515 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:06.990610 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:06.990846 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 17:35:06.990912 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":210,"threads_started":1}
I20260923 17:35:06.992424 26298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:06.985312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:42495 (local address 127.20.240.65:57093)
0923 17:35:06.986337 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:06.986384 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:06.988452 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:06.989181 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:06.989196 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:06.989333 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:06.990046 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:35:06.990050 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:06.990103 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:35:06.990222 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:06.990601 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:06.990611 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:06.990618 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:35:06.990890 (+ 272us) client_negotiation.cc:240] Negotiation successful
0923 17:35:06.991197 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":292,"threads_started":1}
I20260923 17:35:06.994004 26295 heartbeater.cc:347] Connected to a master server at 127.20.240.126:42495
I20260923 17:35:06.994300 26295 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:06.994903 26295 heartbeater.cc:511] Master 127.20.240.126:42495 requested a full tablet report, sending...
I20260923 17:35:06.997262 26109 ts_manager.cc:194] Registered new tserver with Master: b690fe42b7f140a4b36f6d0c04a4ae54 (127.20.240.65:36403)
W20260923 17:35:07.312908 26299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:07.313202 26299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:07.313318 26299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:07.320823 26299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:07.320935 26299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:07.321077 26299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:35:07.330283 26299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:46735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:07.331591 26299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:07.333391 26299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:07.347066 26305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:07.347055 26304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:07.347464 26307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:07.347874 26299 server_base.cc:1034] running on GCE node
I20260923 17:35:07.348515 26299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:07.349500 26299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:07.350735 26299 hybrid_clock.cc:648] HybridClock initialized: now 1790184907350678 us; error 45 us; skew 500 ppm
I20260923 17:35:07.351190 26299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:35:07.353588 26299 webserver.cc:460] Webserver started at http://127.20.240.66:44961/ using document root <none> and password file <none>
I20260923 17:35:07.354321 26299 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:07.354429 26299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:07.354745 26299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:07.357761 26299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "a094f59877004d59bbe0e51aa4c497f5"
format_stamp: "Formatted at 2026-09-23 17:35:07 on dist-test-slave-ppjl"
server_key: "87e11d6aaac5c0b6b5da2cd85228d86d"
server_key_iv: "358ff0e49a0150c8b05cd8317d8c19dc"
server_key_version: "encryptionkey@0"
I20260923 17:35:07.358577 26299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "a094f59877004d59bbe0e51aa4c497f5"
format_stamp: "Formatted at 2026-09-23 17:35:07 on dist-test-slave-ppjl"
server_key: "87e11d6aaac5c0b6b5da2cd85228d86d"
server_key_iv: "358ff0e49a0150c8b05cd8317d8c19dc"
server_key_version: "encryptionkey@0"
I20260923 17:35:07.364004 26299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260923 17:35:07.367704 26313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:07.369136 26299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 17:35:07.369285 26299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "a094f59877004d59bbe0e51aa4c497f5"
format_stamp: "Formatted at 2026-09-23 17:35:07 on dist-test-slave-ppjl"
server_key: "87e11d6aaac5c0b6b5da2cd85228d86d"
server_key_iv: "358ff0e49a0150c8b05cd8317d8c19dc"
server_key_version: "encryptionkey@0"
I20260923 17:35:07.369491 26299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:07.375416 26299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:07.376240 26299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:07.376487 26299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:07.377408 26299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:07.378990 26299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:07.379087 26299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:07.379221 26299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:07.379293 26299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:07.407087 26299 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:45299
I20260923 17:35:07.407138 26425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:45299 every 8 connection(s)
I20260923 17:35:07.408392 26299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:35:07.409325 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 26299
I20260923 17:35:07.409494 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:35:07.410012 21441 external_mini_cluster.cc:1428] Setting key adcb374080efea9c9ff006f27802f247
I20260923 17:35:07.412813 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42495
--builtin_ntp_servers=127.20.240.84:46735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 17:35:07.417260 26297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:07.411061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:54997 (local address 127.20.240.126:42495)
0923 17:35:07.411332 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:07.411338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:07.413001 (+ 1663us) server_negotiation.cc:407] Connection header received
0923 17:35:07.414843 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:07.414850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:07.414923 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:07.415013 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:07.416514 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:07.416519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:07.416522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:07.416539 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:07.416570 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:07.416582 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:07.416692 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:07.416890 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 17:35:07.416964 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:35:07.418329 26428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:07.411720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:42495 (local address 127.20.240.66:54997)
0923 17:35:07.412840 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:07.412887 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:07.414521 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:07.415371 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:07.415384 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:07.415529 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:07.416190 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:35:07.416192 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:07.416236 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:35:07.416334 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:07.416679 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:07.416690 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:07.416699 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:35:07.416965 (+ 266us) client_negotiation.cc:240] Negotiation successful
0923 17:35:07.417326 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":343,"threads_started":1}
I20260923 17:35:07.419838 26426 heartbeater.cc:347] Connected to a master server at 127.20.240.126:42495
I20260923 17:35:07.420135 26426 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:07.420749 26426 heartbeater.cc:511] Master 127.20.240.126:42495 requested a full tablet report, sending...
I20260923 17:35:07.422163 26109 ts_manager.cc:194] Registered new tserver with Master: a094f59877004d59bbe0e51aa4c497f5 (127.20.240.66:45299)
W20260923 17:35:07.737330 26429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:07.737625 26429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:07.737747 26429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:07.745471 26429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:07.745580 26429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:07.745728 26429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:35:07.755178 26429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:46735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42495
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:07.756466 26429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:07.758224 26429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:07.771404 26434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:07.771382 26435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:07.771430 26437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:07.772740 26429 server_base.cc:1034] running on GCE node
I20260923 17:35:07.773388 26429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:07.774415 26429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:07.775645 26429 hybrid_clock.cc:648] HybridClock initialized: now 1790184907775565 us; error 63 us; skew 500 ppm
I20260923 17:35:07.775961 26429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:35:07.778357 26429 webserver.cc:460] Webserver started at http://127.20.240.67:45359/ using document root <none> and password file <none>
I20260923 17:35:07.779048 26429 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:07.779140 26429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:07.779506 26429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:07.782464 26429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "b77c2f15eb22416ab8356f14404a7ebb"
format_stamp: "Formatted at 2026-09-23 17:35:07 on dist-test-slave-ppjl"
server_key: "4b35c5ba4f2202fb3f2bb35b3a55ccfa"
server_key_iv: "18e0b93dcc9e161d1a7aa5293cff0de0"
server_key_version: "encryptionkey@0"
I20260923 17:35:07.783367 26429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "b77c2f15eb22416ab8356f14404a7ebb"
format_stamp: "Formatted at 2026-09-23 17:35:07 on dist-test-slave-ppjl"
server_key: "4b35c5ba4f2202fb3f2bb35b3a55ccfa"
server_key_iv: "18e0b93dcc9e161d1a7aa5293cff0de0"
server_key_version: "encryptionkey@0"
I20260923 17:35:07.788825 26429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:35:07.792464 26443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:07.793841 26429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:35:07.793973 26429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "b77c2f15eb22416ab8356f14404a7ebb"
format_stamp: "Formatted at 2026-09-23 17:35:07 on dist-test-slave-ppjl"
server_key: "4b35c5ba4f2202fb3f2bb35b3a55ccfa"
server_key_iv: "18e0b93dcc9e161d1a7aa5293cff0de0"
server_key_version: "encryptionkey@0"
I20260923 17:35:07.794191 26429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:07.799885 26429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:07.800711 26429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:07.800984 26429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:07.801885 26429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:07.803417 26429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:07.803503 26429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:07.803586 26429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:07.803637 26429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:07.835434 26429 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:38051
I20260923 17:35:07.835507 26555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:38051 every 8 connection(s)
I20260923 17:35:07.836757 26429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:35:07.841862 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 26429
I20260923 17:35:07.842059 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:35:07.842643 21441 external_mini_cluster.cc:1428] Setting key 611fef90650828d115019971107fe6d0
I20260923 17:35:07.846318 26297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:07.839377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:56761 (local address 127.20.240.126:42495)
0923 17:35:07.839549 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:07.839570 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:07.841151 (+ 1581us) server_negotiation.cc:407] Connection header received
0923 17:35:07.843533 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:07.843539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:07.843617 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:07.843714 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:07.845506 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:07.845513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:07.845518 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:07.845541 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:07.845581 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:07.845595 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:07.845725 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:07.845918 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 17:35:07.845987 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:35:07.847453 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:07.839896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:42495 (local address 127.20.240.67:56761)
0923 17:35:07.840865 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:07.840911 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:07.843146 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:07.843987 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:07.843998 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:07.844118 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:07.845129 (+ 1011us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:35:07.845134 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:07.845188 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:35:07.845314 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:07.845691 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:07.845704 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:07.845713 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:35:07.846001 (+ 288us) client_negotiation.cc:240] Negotiation successful
0923 17:35:07.846280 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":306,"threads_started":1}
I20260923 17:35:07.848773 26556 heartbeater.cc:347] Connected to a master server at 127.20.240.126:42495
I20260923 17:35:07.849069 26556 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:07.849694 26556 heartbeater.cc:511] Master 127.20.240.126:42495 requested a full tablet report, sending...
I20260923 17:35:07.851125 26109 ts_manager.cc:194] Registered new tserver with Master: b77c2f15eb22416ab8356f14404a7ebb (127.20.240.67:38051)
I20260923 17:35:07.858279 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:35:07.870340 26297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:07.862657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.20.240.126:42495)
0923 17:35:07.862817 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:07.862822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:07.863389 (+ 567us) server_negotiation.cc:407] Connection header received
0923 17:35:07.863576 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:07.863582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:07.863641 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:07.863720 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:07.869667 (+ 5947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:07.869672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:07.869676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:07.869696 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:07.869736 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:07.869748 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:07.869892 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:07.869982 (+ 90us) server_negotiation.cc:299] Negotiation successful
0923 17:35:07.870054 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:35:07.874708 26109 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:35:07.881621 26109 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:35:07.915196 26569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:07.894361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55474 (local address 127.20.240.67:38051)
0923 17:35:07.911838 (+ 17477us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:07.911845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:07.911879 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:35:07.911944 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:07.911951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:07.912131 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:07.912266 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:07.914135 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:07.914144 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:07.914160 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:07.914181 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:07.914254 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:07.914289 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:07.914460 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:07.914699 (+ 239us) server_negotiation.cc:299] Negotiation successful
0923 17:35:07.914815 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16642,"thread_start_us":16466,"threads_started":1}
I20260923 17:35:07.915308 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:07.894257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:38051 (local address 127.0.0.1:55474)
0923 17:35:07.906496 (+ 12239us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:07.906565 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:07.906765 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:07.912313 (+ 5548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:07.912322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:07.912423 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:07.913791 (+ 1368us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:35:07.913795 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:07.913828 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:35:07.913927 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:07.914508 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:07.914518 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:07.914524 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:35:07.914676 (+ 152us) client_negotiation.cc:240] Negotiation successful
0923 17:35:07.914791 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11836,"thread_start_us":17220,"threads_started":1}
I20260923 17:35:07.915988 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:07.894344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:36403 (local address 127.0.0.1:50510)
0923 17:35:07.910402 (+ 16058us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:07.910428 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:07.910546 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:07.913230 (+ 2684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:07.913234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:07.913259 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:07.914291 (+ 1032us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:35:07.914295 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:07.914310 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:35:07.914351 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:07.914876 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:07.914880 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:07.914885 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:35:07.915035 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 17:35:07.915101 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15875,"thread_start_us":11589,"threads_started":1}
I20260923 17:35:07.916014 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:07.894622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50510 (local address 127.20.240.65:36403)
0923 17:35:07.911561 (+ 16939us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:07.911571 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:07.911604 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:35:07.911686 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:07.911695 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:07.911986 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:07.912152 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:07.914540 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:07.914550 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:07.914567 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:07.914595 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:07.914651 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:07.914671 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:07.914822 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:07.915614 (+ 792us) server_negotiation.cc:299] Negotiation successful
0923 17:35:07.915719 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16798,"thread_start_us":16684,"threads_started":1}
I20260923 17:35:07.916658 26573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:07.896708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.20.240.66:45299)
0923 17:35:07.911534 (+ 14826us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:07.911546 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:07.912114 (+ 568us) server_negotiation.cc:407] Connection header received
0923 17:35:07.912371 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:07.912379 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:07.913030 (+ 651us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:07.913181 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:07.915087 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:07.915096 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:07.915108 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:07.915126 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:07.915211 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:07.915239 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:07.915389 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:07.916202 (+ 813us) server_negotiation.cc:299] Negotiation successful
0923 17:35:07.916303 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14650,"thread_start_us":14527,"threads_started":1}
I20260923 17:35:07.916038 26572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:07.896177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:45299 (local address 127.0.0.1:57152)
0923 17:35:07.911979 (+ 15802us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:07.911994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:07.912107 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:07.913596 (+ 1489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:07.913601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:07.913846 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:07.914833 (+ 987us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:35:07.914837 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:07.914852 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:35:07.914894 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:07.915444 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:07.915448 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:07.915452 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 17:35:07.915567 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 17:35:07.915649 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15685,"thread_start_us":15546,"threads_started":1}
I20260923 17:35:07.919672 26491 tablet_service.cc:1467] Processing CreateTablet for tablet d2c07cfe572746f3816c8da891b8aae4 (DEFAULT_TABLE table=test-table [id=a617fadc1e9643a3a0826fb4dca0a41e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:07.920156 26361 tablet_service.cc:1467] Processing CreateTablet for tablet d2c07cfe572746f3816c8da891b8aae4 (DEFAULT_TABLE table=test-table [id=a617fadc1e9643a3a0826fb4dca0a41e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:07.920365 26230 tablet_service.cc:1467] Processing CreateTablet for tablet d2c07cfe572746f3816c8da891b8aae4 (DEFAULT_TABLE table=test-table [id=a617fadc1e9643a3a0826fb4dca0a41e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:07.921241 26491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c07cfe572746f3816c8da891b8aae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:07.921375 26361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c07cfe572746f3816c8da891b8aae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:07.921602 26230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c07cfe572746f3816c8da891b8aae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:07.933501 26574 tablet_bootstrap.cc:492] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54: Bootstrap starting.
I20260923 17:35:07.935925 26575 tablet_bootstrap.cc:492] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb: Bootstrap starting.
I20260923 17:35:07.937480 26576 tablet_bootstrap.cc:492] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5: Bootstrap starting.
I20260923 17:35:07.937701 26574 tablet_bootstrap.cc:654] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:07.939085 26574 log.cc:824] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:07.939747 26575 tablet_bootstrap.cc:654] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:07.940944 26576 tablet_bootstrap.cc:654] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:07.941104 26575 log.cc:824] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:07.942171 26576 log.cc:824] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:07.945299 26574 tablet_bootstrap.cc:492] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54: No bootstrap required, opened a new log
I20260923 17:35:07.945457 26576 tablet_bootstrap.cc:492] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5: No bootstrap required, opened a new log
I20260923 17:35:07.945591 26574 ts_tablet_manager.cc:1397] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260923 17:35:07.945631 26575 tablet_bootstrap.cc:492] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb: No bootstrap required, opened a new log
I20260923 17:35:07.945739 26576 ts_tablet_manager.cc:1397] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 17:35:07.945868 26575 ts_tablet_manager.cc:1397] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260923 17:35:07.953938 26574 raft_consensus.cc:348] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b77c2f15eb22416ab8356f14404a7ebb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38051 } } peers { permanent_uuid: "b690fe42b7f140a4b36f6d0c04a4ae54" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36403 } } peers { permanent_uuid: "a094f59877004d59bbe0e51aa4c497f5" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45299 } }
I20260923 17:35:07.954373 26574 raft_consensus.cc:374] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:07.954476 26574 raft_consensus.cc:729] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b690fe42b7f140a4b36f6d0c04a4ae54, State: Initialized, Role: FOLLOWER
I20260923 17:35:07.955143 26574 consensus_queue.cc:260] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b77c2f15eb22416ab8356f14404a7ebb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38051 } } peers { permanent_uuid: "b690fe42b7f140a4b36f6d0c04a4ae54" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36403 } } peers { permanent_uuid: "a094f59877004d59bbe0e51aa4c497f5" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45299 } }
I20260923 17:35:07.956372 26295 heartbeater.cc:503] Master 127.20.240.126:42495 was elected leader, sending a full tablet report...
I20260923 17:35:07.956501 26575 raft_consensus.cc:348] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b77c2f15eb22416ab8356f14404a7ebb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38051 } } peers { permanent_uuid: "b690fe42b7f140a4b36f6d0c04a4ae54" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36403 } } peers { permanent_uuid: "a094f59877004d59bbe0e51aa4c497f5" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45299 } }
I20260923 17:35:07.956939 26574 ts_tablet_manager.cc:1428] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260923 17:35:07.956979 26575 raft_consensus.cc:374] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:07.957085 26575 raft_consensus.cc:729] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b77c2f15eb22416ab8356f14404a7ebb, State: Initialized, Role: FOLLOWER
I20260923 17:35:07.957741 26575 consensus_queue.cc:260] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b77c2f15eb22416ab8356f14404a7ebb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38051 } } peers { permanent_uuid: "b690fe42b7f140a4b36f6d0c04a4ae54" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36403 } } peers { permanent_uuid: "a094f59877004d59bbe0e51aa4c497f5" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45299 } }
I20260923 17:35:07.958091 26576 raft_consensus.cc:348] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b77c2f15eb22416ab8356f14404a7ebb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38051 } } peers { permanent_uuid: "b690fe42b7f140a4b36f6d0c04a4ae54" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36403 } } peers { permanent_uuid: "a094f59877004d59bbe0e51aa4c497f5" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45299 } }
I20260923 17:35:07.958612 26576 raft_consensus.cc:374] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:07.958712 26576 raft_consensus.cc:729] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a094f59877004d59bbe0e51aa4c497f5, State: Initialized, Role: FOLLOWER
I20260923 17:35:07.959043 26556 heartbeater.cc:503] Master 127.20.240.126:42495 was elected leader, sending a full tablet report...
I20260923 17:35:07.959679 26575 ts_tablet_manager.cc:1428] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260923 17:35:07.959506 26576 consensus_queue.cc:260] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b77c2f15eb22416ab8356f14404a7ebb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38051 } } peers { permanent_uuid: "b690fe42b7f140a4b36f6d0c04a4ae54" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36403 } } peers { permanent_uuid: "a094f59877004d59bbe0e51aa4c497f5" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45299 } }
I20260923 17:35:07.962337 26426 heartbeater.cc:503] Master 127.20.240.126:42495 was elected leader, sending a full tablet report...
I20260923 17:35:07.962949 26576 ts_tablet_manager.cc:1428] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5: Time spent starting tablet: real 0.017s user 0.007s sys 0.009s
W20260923 17:35:07.984310 26296 tablet.cc:2369] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:35:08.088414 26557 tablet.cc:2369] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:35:08.105660 26580 raft_consensus.cc:482] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:35:08.105973 26580 raft_consensus.cc:504] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b77c2f15eb22416ab8356f14404a7ebb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38051 } } peers { permanent_uuid: "b690fe42b7f140a4b36f6d0c04a4ae54" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36403 } } peers { permanent_uuid: "a094f59877004d59bbe0e51aa4c497f5" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45299 } }
I20260923 17:35:08.107753 26580 leader_election.cc:288] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b77c2f15eb22416ab8356f14404a7ebb (127.20.240.67:38051), a094f59877004d59bbe0e51aa4c497f5 (127.20.240.66:45299)
I20260923 17:35:08.111297 26569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:08.108159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:58219 (local address 127.20.240.67:38051)
0923 17:35:08.108358 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:08.108363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:08.109318 (+ 955us) server_negotiation.cc:407] Connection header received
0923 17:35:08.109445 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:08.109449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:08.109498 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:08.109576 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:08.110695 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:08.110699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:08.110702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:08.110719 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:08.110753 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:08.110765 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:08.110879 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:08.110954 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 17:35:08.111019 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 17:35:08.111393 26573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:08.108224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:37451 (local address 127.20.240.66:45299)
0923 17:35:08.108433 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:08.108438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:08.109241 (+ 803us) server_negotiation.cc:407] Connection header received
0923 17:35:08.109391 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:08.109396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:08.109450 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:08.109531 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:08.110777 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:08.110781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:08.110784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:08.110800 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:08.110830 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:08.110842 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:08.110952 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:08.111061 (+ 109us) server_negotiation.cc:299] Negotiation successful
0923 17:35:08.111125 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 17:35:08.111863 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:08.108311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:38051 (local address 127.20.240.65:58219)
0923 17:35:08.109206 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:08.109221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:08.109329 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:08.109617 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:08.109621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:08.109642 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:08.110490 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:35:08.110492 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:08.110510 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:35:08.110550 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:08.110851 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:08.110856 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:08.110860 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 17:35:08.110994 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 17:35:08.111043 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":494,"threads_started":1}
I20260923 17:35:08.111887 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:08.108179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:45299 (local address 127.20.240.65:37451)
0923 17:35:08.109131 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:08.109147 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:08.109271 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:08.109576 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:08.109580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:08.109602 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:08.110560 (+ 958us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:35:08.110563 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:08.110577 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:35:08.110613 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:08.110927 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:08.110931 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:08.110936 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:35:08.111088 (+ 152us) client_negotiation.cc:240] Negotiation successful
0923 17:35:08.111133 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":414,"threads_started":1}
I20260923 17:35:08.112769 26381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c07cfe572746f3816c8da891b8aae4" candidate_uuid: "b690fe42b7f140a4b36f6d0c04a4ae54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a094f59877004d59bbe0e51aa4c497f5" is_pre_election: true
I20260923 17:35:08.112788 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c07cfe572746f3816c8da891b8aae4" candidate_uuid: "b690fe42b7f140a4b36f6d0c04a4ae54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b77c2f15eb22416ab8356f14404a7ebb" is_pre_election: true
I20260923 17:35:08.113214 26381 raft_consensus.cc:2454] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b690fe42b7f140a4b36f6d0c04a4ae54 in term 0.
I20260923 17:35:08.113214 26511 raft_consensus.cc:2454] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b690fe42b7f140a4b36f6d0c04a4ae54 in term 0.
I20260923 17:35:08.113785 26183 leader_election.cc:302] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a094f59877004d59bbe0e51aa4c497f5, b690fe42b7f140a4b36f6d0c04a4ae54; no voters:
I20260923 17:35:08.114130 26580 raft_consensus.cc:2781] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:35:08.114297 26580 raft_consensus.cc:482] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:35:08.114398 26580 raft_consensus.cc:3037] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:08.116643 26580 raft_consensus.cc:504] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b77c2f15eb22416ab8356f14404a7ebb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38051 } } peers { permanent_uuid: "b690fe42b7f140a4b36f6d0c04a4ae54" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36403 } } peers { permanent_uuid: "a094f59877004d59bbe0e51aa4c497f5" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45299 } }
I20260923 17:35:08.117223 26580 leader_election.cc:288] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [CANDIDATE]: Term 1 election: Requested vote from peers b77c2f15eb22416ab8356f14404a7ebb (127.20.240.67:38051), a094f59877004d59bbe0e51aa4c497f5 (127.20.240.66:45299)
I20260923 17:35:08.117671 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c07cfe572746f3816c8da891b8aae4" candidate_uuid: "b690fe42b7f140a4b36f6d0c04a4ae54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b77c2f15eb22416ab8356f14404a7ebb"
I20260923 17:35:08.117703 26381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c07cfe572746f3816c8da891b8aae4" candidate_uuid: "b690fe42b7f140a4b36f6d0c04a4ae54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a094f59877004d59bbe0e51aa4c497f5"
I20260923 17:35:08.117864 26511 raft_consensus.cc:3037] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:08.117877 26381 raft_consensus.cc:3037] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:08.120066 26381 raft_consensus.cc:2454] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b690fe42b7f140a4b36f6d0c04a4ae54 in term 1.
I20260923 17:35:08.120069 26511 raft_consensus.cc:2454] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b690fe42b7f140a4b36f6d0c04a4ae54 in term 1.
I20260923 17:35:08.120472 26183 leader_election.cc:302] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a094f59877004d59bbe0e51aa4c497f5, b690fe42b7f140a4b36f6d0c04a4ae54; no voters:
I20260923 17:35:08.120762 26580 raft_consensus.cc:2781] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:08.121263 26580 raft_consensus.cc:686] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [term 1 LEADER]: Becoming Leader. State: Replica: b690fe42b7f140a4b36f6d0c04a4ae54, State: Running, Role: LEADER
I20260923 17:35:08.121721 26580 consensus_queue.cc:237] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b77c2f15eb22416ab8356f14404a7ebb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38051 } } peers { permanent_uuid: "b690fe42b7f140a4b36f6d0c04a4ae54" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36403 } } peers { permanent_uuid: "a094f59877004d59bbe0e51aa4c497f5" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45299 } }
I20260923 17:35:08.126518 26109 catalog_manager.cc:5495] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 reported cstate change: term changed from 0 to 1, leader changed from <none> to b690fe42b7f140a4b36f6d0c04a4ae54 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "b690fe42b7f140a4b36f6d0c04a4ae54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b77c2f15eb22416ab8356f14404a7ebb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b690fe42b7f140a4b36f6d0c04a4ae54" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a094f59877004d59bbe0e51aa4c497f5" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45299 } health_report { overall_health: UNKNOWN } } }
I20260923 17:35:08.158349 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:08.155996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50518 (local address 127.20.240.65:36403)
0923 17:35:08.156167 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:08.156174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:08.156252 (+ 78us) server_negotiation.cc:407] Connection header received
0923 17:35:08.156511 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:08.156515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:08.156597 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:08.156682 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:08.157657 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:08.157662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:08.157665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:08.157683 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:08.157721 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:08.157735 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:08.157849 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:08.157969 (+ 120us) server_negotiation.cc:299] Negotiation successful
0923 17:35:08.158038 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260923 17:35:08.160456 26427 tablet.cc:2369] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:35:08.166779 26381 raft_consensus.cc:1261] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5 [term 1 FOLLOWER]: Refusing update from remote peer b690fe42b7f140a4b36f6d0c04a4ae54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:08.166791 26511 raft_consensus.cc:1261] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb [term 1 FOLLOWER]: Refusing update from remote peer b690fe42b7f140a4b36f6d0c04a4ae54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:08.167766 26580 consensus_queue.cc:1035] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b77c2f15eb22416ab8356f14404a7ebb" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 38051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:08.168085 26585 consensus_queue.cc:1035] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a094f59877004d59bbe0e51aa4c497f5" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:08.181207 26589 mvcc.cc:205] Tried to move back new op lower bound from 7332597383841812480 to 7332597383670263808. Current Snapshot: MvccSnapshot[applied={T|T < 7332597383841812480}]
I20260923 17:35:08.200563 26108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:08.200888 26108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:08.204778 26108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 460e973086164bf7ab83b5401bd9817c: Sending DeleteTablet for 3 replicas of tablet d2c07cfe572746f3816c8da891b8aae4
I20260923 17:35:08.205987 26491 tablet_service.cc:1514] Processing DeleteTablet for tablet d2c07cfe572746f3816c8da891b8aae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:08 UTC) from {username='slave'} at 127.0.0.1:55474
I20260923 17:35:08.205986 26361 tablet_service.cc:1514] Processing DeleteTablet for tablet d2c07cfe572746f3816c8da891b8aae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:08 UTC) from {username='slave'} at 127.0.0.1:57152
I20260923 17:35:08.206035 26230 tablet_service.cc:1514] Processing DeleteTablet for tablet d2c07cfe572746f3816c8da891b8aae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:08 UTC) from {username='slave'} at 127.0.0.1:50510
I20260923 17:35:08.210438 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 26168
I20260923 17:35:08.219260 26599 tablet_replica.cc:330] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb: stopping tablet replica
I20260923 17:35:08.219923 26600 tablet_replica.cc:330] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5: stopping tablet replica
I20260923 17:35:08.220525 26600 raft_consensus.cc:2229] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:08.221136 26599 raft_consensus.cc:2229] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:08.221446 26601 tablet_replica.cc:330] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54: stopping tablet replica
I20260923 17:35:08.221827 26601 raft_consensus.cc:2229] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:35:08.221926 26571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:08.211943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50528 (local address 127.20.240.65:36403)
0923 17:35:08.212133 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:08.212139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:08.214615 (+ 2476us) server_negotiation.cc:407] Connection header received
0923 17:35:08.214866 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:08.214873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:08.214944 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:08.215061 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:08.220996 (+ 5935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:08.221003 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:08.221008 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:08.221040 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:08.221083 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:08.221101 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:08.221260 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:08.221530 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 17:35:08.221600 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:35:08.222316 26600 raft_consensus.cc:2258] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:08.222464 26601 raft_consensus.cc:2258] T d2c07cfe572746f3816c8da891b8aae4 P b690fe42b7f140a4b36f6d0c04a4ae54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:08.223026 26290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:08.222690 26599 raft_consensus.cc:2258] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:08.225735 26600 ts_tablet_manager.cc:1905] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:08.225760 26599 ts_tablet_manager.cc:1905] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:08.232844 26600 ts_tablet_manager.cc:1918] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:08.232844 26599 ts_tablet_manager.cc:1918] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:08.233017 26599 log.cc:1196] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/d2c07cfe572746f3816c8da891b8aae4
I20260923 17:35:08.233017 26600 log.cc:1196] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/d2c07cfe572746f3816c8da891b8aae4
I20260923 17:35:08.233525 26600 ts_tablet_manager.cc:1939] T d2c07cfe572746f3816c8da891b8aae4 P a094f59877004d59bbe0e51aa4c497f5: Deleting consensus metadata
I20260923 17:35:08.233525 26599 ts_tablet_manager.cc:1939] T d2c07cfe572746f3816c8da891b8aae4 P b77c2f15eb22416ab8356f14404a7ebb: Deleting consensus metadata
I20260923 17:35:08.235426 26093 catalog_manager.cc:4841] TS a094f59877004d59bbe0e51aa4c497f5 (127.20.240.66:45299): tablet d2c07cfe572746f3816c8da891b8aae4 (table test-table [id=a617fadc1e9643a3a0826fb4dca0a41e]) successfully deleted
I20260923 17:35:08.235381 26095 catalog_manager.cc:4841] TS b77c2f15eb22416ab8356f14404a7ebb (127.20.240.67:38051): tablet d2c07cfe572746f3816c8da891b8aae4 (table test-table [id=a617fadc1e9643a3a0826fb4dca0a41e]) successfully deleted
I20260923 17:35:08.318688 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 26168
I20260923 17:35:08.332456 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 26299
W20260923 17:35:08.332338 26096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.240.65:36403 (error 108)
I20260923 17:35:08.334415 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:08.333904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:36403 (local address 127.0.0.1:50544)
0923 17:35:08.334087 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:08.334242 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 17:35:08.334668 26096 catalog_manager.cc:4584] TS b690fe42b7f140a4b36f6d0c04a4ae54 (127.20.240.65:36403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2c07cfe572746f3816c8da891b8aae4: Network error: Client connection negotiation failed: client connection to 127.20.240.65:36403: connect: Connection refused (error 111)
I20260923 17:35:08.343397 26573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:08.333398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.20.240.66:45299)
0923 17:35:08.333564 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:08.333569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:08.333634 (+ 65us) server_negotiation.cc:407] Connection header received
0923 17:35:08.333854 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:08.333860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:08.333965 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:08.334177 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:08.342643 (+ 8466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:08.342648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:08.342651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:08.342672 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:08.342711 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:08.342725 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:08.342852 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:08.342964 (+ 112us) server_negotiation.cc:299] Negotiation successful
0923 17:35:08.343034 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:35:08.344159 26421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:08.388334 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:08.387744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:36403 (local address 127.0.0.1:50556)
0923 17:35:08.388018 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:08.388131 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 17:35:08.389389 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:08.388985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:36403 (local address 127.0.0.1:50572)
0923 17:35:08.389143 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:08.389240 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 17:35:08.432554 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 26299
I20260923 17:35:08.446559 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 26429
I20260923 17:35:08.449683 26569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:08.447342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55486 (local address 127.20.240.67:38051)
0923 17:35:08.447523 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:08.447527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:08.447572 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:35:08.447781 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:08.447785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:08.447866 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:08.447978 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:08.448890 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:08.448895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:08.448898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:08.448913 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:08.448946 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:08.448958 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:08.449074 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:08.449283 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 17:35:08.449344 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:35:08.450014 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:08.449522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:36403 (local address 127.0.0.1:50588)
0923 17:35:08.449728 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:08.449855 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260923 17:35:08.450352 26551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:08.451256 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:08.450801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:36403 (local address 127.0.0.1:50602)
0923 17:35:08.450971 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:08.451068 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 17:35:08.532155 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:08.531489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:36403 (local address 127.0.0.1:50604)
0923 17:35:08.531755 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:08.531912 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260923 17:35:08.533465 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:08.532937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:36403 (local address 127.0.0.1:50618)
0923 17:35:08.533193 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:08.533297 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:36403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260923 17:35:08.540223 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 26429
I20260923 17:35:08.553196 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 26078
I20260923 17:35:08.554523 26139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:08.661607 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 26078
2026-09-23T17:35:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2692 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 17:35:08.698587 21441 mini_oidc.cc:152] Starting JWKS server
I20260923 17:35:08.701161 21441 webserver.cc:460] Webserver started at https://127.0.0.1:34659/ using document root <none> and password file <none>
I20260923 17:35:08.701468 21441 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 17:35:08.702131 21441 webserver.cc:460] Webserver started at http://127.0.0.1:32981/ using document root <none> and password file <none>
2026-09-23T17:35:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:35:08Z Disabled control of system clock
I20260923 17:35:08.727670 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:39687
--webserver_interface=127.20.240.126
--webserver_port=0
--builtin_ntp_servers=127.20.240.84:44361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:39687
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34659/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/testchainca.cert with env {}
W20260923 17:35:09.052711 26613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:09.053126 26613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:09.053274 26613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:09.061082 26613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:35:09.061247 26613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:09.061311 26613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:09.061398 26613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:35:09.061467 26613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:35:09.061560 26613 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:35:09.061611 26613 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34659/jwks.json
I20260923 17:35:09.072094 26613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:39687
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:39687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34659/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:09.073468 26613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:09.074800 26613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:09.086594 26621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:09.086819 26619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:09.086766 26618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:09.087924 26613 server_base.cc:1034] running on GCE node
I20260923 17:35:09.088670 26613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:09.090144 26613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:09.091351 26613 hybrid_clock.cc:648] HybridClock initialized: now 1790184909091299 us; error 44 us; skew 500 ppm
I20260923 17:35:09.091763 26613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:35:09.094162 26613 webserver.cc:460] Webserver started at http://127.20.240.126:41405/ using document root <none> and password file <none>
I20260923 17:35:09.094893 26613 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:09.095009 26613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:09.095480 26613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:09.098434 26613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "90147f30b8344545beba28770e2ac46d"
format_stamp: "Formatted at 2026-09-23 17:35:09 on dist-test-slave-ppjl"
server_key: "1b4fb63cdb7fca31b7f5a64b5d01f683"
server_key_iv: "2e34b28de358b3ad5a43cbbe2dffdd67"
server_key_version: "encryptionkey@0"
I20260923 17:35:09.099251 26613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "90147f30b8344545beba28770e2ac46d"
format_stamp: "Formatted at 2026-09-23 17:35:09 on dist-test-slave-ppjl"
server_key: "1b4fb63cdb7fca31b7f5a64b5d01f683"
server_key_iv: "2e34b28de358b3ad5a43cbbe2dffdd67"
server_key_version: "encryptionkey@0"
I20260923 17:35:09.104687 26613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:35:09.108361 26627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:09.109827 26613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:35:09.110006 26613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "90147f30b8344545beba28770e2ac46d"
format_stamp: "Formatted at 2026-09-23 17:35:09 on dist-test-slave-ppjl"
server_key: "1b4fb63cdb7fca31b7f5a64b5d01f683"
server_key_iv: "2e34b28de358b3ad5a43cbbe2dffdd67"
server_key_version: "encryptionkey@0"
I20260923 17:35:09.110222 26613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:09.124784 26613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:09.125610 26613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:09.125897 26613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:09.143733 26613 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:39687
I20260923 17:35:09.143744 26678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:39687 every 8 connection(s)
I20260923 17:35:09.145123 26613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:35:09.145498 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 26613
I20260923 17:35:09.145686 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:35:09.146405 21441 external_mini_cluster.cc:1428] Setting key 31659c16f155e01b9ddf8c61772bdca9
I20260923 17:35:09.149456 26679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:09.161464 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:09.148457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58340 (local address 127.20.240.126:39687)
0923 17:35:09.149438 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:09.149460 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:09.149502 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:35:09.151141 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:09.151196 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:09.151721 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:09.152091 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:09.152762 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:09.154206 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:09.156229 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:09.157011 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:09.157268 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:09.158657 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:09.158666 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:09.158686 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:09.159468 (+ 782us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:09.159568 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:09.159603 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:09.159715 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:09.159988 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 17:35:09.160358 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":338,"threads_started":1}
I20260923 17:35:09.162187 26679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d: Bootstrap starting.
I20260923 17:35:09.165696 26679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:09.165756 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:35:09.166648 26679 log.cc:824] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:09.169749 26679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d: No bootstrap required, opened a new log
I20260923 17:35:09.177461 26679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90147f30b8344545beba28770e2ac46d" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39687 } }
I20260923 17:35:09.177873 26679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:09.177974 26679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90147f30b8344545beba28770e2ac46d, State: Initialized, Role: FOLLOWER
I20260923 17:35:09.178591 26679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90147f30b8344545beba28770e2ac46d" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39687 } }
I20260923 17:35:09.178833 26679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:35:09.178967 26679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:35:09.179150 26679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:09.181571 26679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90147f30b8344545beba28770e2ac46d" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39687 } }
I20260923 17:35:09.182232 26679 leader_election.cc:302] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90147f30b8344545beba28770e2ac46d; no voters:
I20260923 17:35:09.182871 26679 leader_election.cc:288] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:35:09.183009 26684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:09.183692 26684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [term 1 LEADER]: Becoming Leader. State: Replica: 90147f30b8344545beba28770e2ac46d, State: Running, Role: LEADER
I20260923 17:35:09.184253 26684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90147f30b8344545beba28770e2ac46d" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39687 } }
I20260923 17:35:09.185076 26679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:35:09.186964 26685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90147f30b8344545beba28770e2ac46d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90147f30b8344545beba28770e2ac46d" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39687 } } }
I20260923 17:35:09.186973 26686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90147f30b8344545beba28770e2ac46d. Latest consensus state: current_term: 1 leader_uuid: "90147f30b8344545beba28770e2ac46d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90147f30b8344545beba28770e2ac46d" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39687 } } }
I20260923 17:35:09.187593 26686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:09.187598 26685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:09.188684 26691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:35:09.194777 26691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:35:09.206169 26691 catalog_manager.cc:1347] Generated new cluster ID: 1bd0c063fa7f4900b64f1fc53e07ecf6
I20260923 17:35:09.206308 26691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:35:09.234009 26691 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:35:09.235459 26691 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:35:09.247808 26691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90147f30b8344545beba28770e2ac46d: Generated new TSK 0
I20260923 17:35:09.248809 26691 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 17:35:09.283239 26709 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 17:35:09.283427 26709 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 17:35:09.284519 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:09.263257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58356 (local address 127.20.240.126:39687)
0923 17:35:09.263441 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:09.263448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:09.263794 (+ 346us) server_negotiation.cc:407] Connection header received
0923 17:35:09.264085 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:09.264091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:09.264185 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:09.264351 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:35:09.264887 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:09.266045 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:09.267497 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:09.268227 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:09.268343 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:09.268915 (+ 572us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:35:09.283810 (+ 14895us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 17:35:09.284182 (+ 372us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58356: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260923 17:35:09.284664 26682 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58356: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 17:35:09.286087 26708 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:09.263065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:39687 (local address 127.0.0.1:58356)
0923 17:35:09.263697 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:09.263713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:09.263860 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:09.264419 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:09.264423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:09.264433 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:35:09.264746 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:09.264753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:09.266167 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:09.266171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:09.267364 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:09.267377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:09.268412 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:09.268417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:09.268537 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:09.268605 (+ 68us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:35:09.284139 (+ 15534us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 17:35:09.284508 (+ 369us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.240.126:39687: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":328,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.20.240.126:39687: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 17:35:09.287889 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 26613
I20260923 17:35:09.289254 26674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:09.383646 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 26613
2026-09-23T17:35:09Z chronyd exiting
I20260923 17:35:09.430449 21441 test_util.cc:175] -----------------------------------------------
I20260923 17:35:09.430608 21441 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184865977475-21441-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (736 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 17:35:09.434504 21441 mini_oidc.cc:152] Starting JWKS server
I20260923 17:35:09.436417 21441 webserver.cc:460] Webserver started at https://127.0.0.1:41171/ using document root <none> and password file <none>
I20260923 17:35:09.436650 21441 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 17:35:09.437443 21441 webserver.cc:460] Webserver started at http://127.0.0.1:44871/ using document root <none> and password file <none>
2026-09-23T17:35:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:35:09Z Disabled control of system clock
I20260923 17:35:09.464484 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:39783
--webserver_interface=127.20.240.126
--webserver_port=0
--builtin_ntp_servers=127.20.240.84:43913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:39783
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41171/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/testchainca.cert with env {}
W20260923 17:35:09.787215 26716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:09.787595 26716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:09.787732 26716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:09.795315 26716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:35:09.795434 26716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:09.795498 26716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:09.795576 26716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:35:09.795645 26716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:35:09.795727 26716 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:35:09.795778 26716 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41171/jwks.json
I20260923 17:35:09.805853 26716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:43913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:39783
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:39783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41171/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:09.807184 26716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:09.808470 26716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:09.820494 26724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:09.820568 26721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:09.820600 26722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:09.821450 26716 server_base.cc:1034] running on GCE node
I20260923 17:35:09.822322 26716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:09.823869 26716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:09.825081 26716 hybrid_clock.cc:648] HybridClock initialized: now 1790184909825043 us; error 34 us; skew 500 ppm
I20260923 17:35:09.825433 26716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:35:09.827759 26716 webserver.cc:460] Webserver started at http://127.20.240.126:44661/ using document root <none> and password file <none>
I20260923 17:35:09.828428 26716 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:09.828538 26716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:09.828856 26716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:09.831892 26716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "52ac114e87cb41ea845541f950a505fd"
format_stamp: "Formatted at 2026-09-23 17:35:09 on dist-test-slave-ppjl"
server_key: "ee261245e3fb9f8abf8b90c36ac0b92a"
server_key_iv: "5d5d0f96e2bafa8fe8f30da43daf4d77"
server_key_version: "encryptionkey@0"
I20260923 17:35:09.832731 26716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "52ac114e87cb41ea845541f950a505fd"
format_stamp: "Formatted at 2026-09-23 17:35:09 on dist-test-slave-ppjl"
server_key: "ee261245e3fb9f8abf8b90c36ac0b92a"
server_key_iv: "5d5d0f96e2bafa8fe8f30da43daf4d77"
server_key_version: "encryptionkey@0"
I20260923 17:35:09.838300 26716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260923 17:35:09.842029 26730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:09.843628 26716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:35:09.843808 26716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "52ac114e87cb41ea845541f950a505fd"
format_stamp: "Formatted at 2026-09-23 17:35:09 on dist-test-slave-ppjl"
server_key: "ee261245e3fb9f8abf8b90c36ac0b92a"
server_key_iv: "5d5d0f96e2bafa8fe8f30da43daf4d77"
server_key_version: "encryptionkey@0"
I20260923 17:35:09.844012 26716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:09.862278 26716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:09.863260 26716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:09.863598 26716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:09.882339 26716 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:39783
I20260923 17:35:09.882305 26781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:39783 every 8 connection(s)
I20260923 17:35:09.883857 26716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:35:09.887537 26782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:09.892967 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 26716
I20260923 17:35:09.893188 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:35:09.893855 21441 external_mini_cluster.cc:1428] Setting key c40c386fc9d1b5a095a1bae940ea9300
I20260923 17:35:09.898335 26782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd: Bootstrap starting.
I20260923 17:35:09.901553 26782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:09.902602 26782 log.cc:824] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:09.905676 26782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd: No bootstrap required, opened a new log
I20260923 17:35:09.908139 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:09.895937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.20.240.126:39783)
0923 17:35:09.896969 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:09.896980 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:09.897029 (+ 49us) server_negotiation.cc:407] Connection header received
0923 17:35:09.898849 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:09.898873 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:09.899418 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:09.899824 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:09.900423 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:09.901813 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:09.903313 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:09.904101 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:09.904302 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:09.905615 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:09.905624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:09.905639 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:09.906287 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:09.906370 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:09.906392 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:09.906507 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:09.906742 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 17:35:09.907074 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":342,"threads_started":1}
I20260923 17:35:09.912055 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:35:09.913599 26782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ac114e87cb41ea845541f950a505fd" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39783 } }
I20260923 17:35:09.913988 26782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:09.914099 26782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ac114e87cb41ea845541f950a505fd, State: Initialized, Role: FOLLOWER
I20260923 17:35:09.914775 26782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ac114e87cb41ea845541f950a505fd" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39783 } }
I20260923 17:35:09.915061 26782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:35:09.915233 26782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:35:09.915431 26782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:09.917585 26782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ac114e87cb41ea845541f950a505fd" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39783 } }
I20260923 17:35:09.918037 26782 leader_election.cc:302] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52ac114e87cb41ea845541f950a505fd; no voters:
I20260923 17:35:09.918579 26782 leader_election.cc:288] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:35:09.918736 26787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:09.919560 26787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [term 1 LEADER]: Becoming Leader. State: Replica: 52ac114e87cb41ea845541f950a505fd, State: Running, Role: LEADER
I20260923 17:35:09.920082 26787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ac114e87cb41ea845541f950a505fd" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39783 } }
I20260923 17:35:09.920908 26782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:35:09.922782 26788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52ac114e87cb41ea845541f950a505fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ac114e87cb41ea845541f950a505fd" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39783 } } }
I20260923 17:35:09.922760 26789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52ac114e87cb41ea845541f950a505fd. Latest consensus state: current_term: 1 leader_uuid: "52ac114e87cb41ea845541f950a505fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ac114e87cb41ea845541f950a505fd" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39783 } } }
I20260923 17:35:09.923192 26789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:09.923192 26788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:09.923985 26793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:35:09.930682 26793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:35:09.940548 26793 catalog_manager.cc:1347] Generated new cluster ID: ca3eb62fa3aa4f3dbd16563d33c9cd20
I20260923 17:35:09.940677 26793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:35:09.977186 26793 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:35:09.978452 26793 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:35:09.986323 26793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52ac114e87cb41ea845541f950a505fd: Generated new TSK 0
I20260923 17:35:09.987179 26793 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 17:35:10.021834 26812 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 17:35:10.021948 26812 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 17:35:10.023324 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:10.003472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.20.240.126:39783)
0923 17:35:10.003657 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:10.003663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:10.004051 (+ 388us) server_negotiation.cc:407] Connection header received
0923 17:35:10.004288 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:10.004292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:10.004365 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:10.004517 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:35:10.005006 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:10.005767 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:10.007199 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:10.007647 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:10.007787 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:10.008274 (+ 487us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:35:10.022575 (+ 14301us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 17:35:10.022986 (+ 411us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37306: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260923 17:35:10.023438 26811 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:10.003372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:39783 (local address 127.0.0.1:37306)
0923 17:35:10.003947 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:10.003962 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:10.004102 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:10.004541 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:10.004545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:10.004558 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:35:10.004830 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:10.004840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:10.005913 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:10.005918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:10.007026 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:10.007039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:10.007813 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:10.007819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:10.007980 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:10.008017 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:35:10.022838 (+ 14821us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 17:35:10.023057 (+ 219us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.240.126:39783: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":318,"threads_started":1}
W20260923 17:35:10.023454 26785 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37306: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260923 17:35:10.024895 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 26716
I20260923 17:35:10.026166 26777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:10.122593 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 26716
2026-09-23T17:35:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (740 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 17:35:10.175515 21441 mini_oidc.cc:152] Starting JWKS server
I20260923 17:35:10.177556 21441 webserver.cc:460] Webserver started at https://127.0.0.1:39051/ using document root <none> and password file <none>
I20260923 17:35:10.177830 21441 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 17:35:10.178556 21441 webserver.cc:460] Webserver started at http://127.0.0.1:41873/ using document root <none> and password file <none>
2026-09-23T17:35:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:35:10Z Disabled control of system clock
I20260923 17:35:10.204411 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:42041
--webserver_interface=127.20.240.126
--webserver_port=0
--builtin_ntp_servers=127.20.240.84:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:42041
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39051/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 17:35:10.533010 26819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:10.533318 26819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:10.533416 26819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:10.540720 26819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:35:10.540861 26819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:10.540899 26819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:10.540956 26819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:35:10.540997 26819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:35:10.541057 26819 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:35:10.541095 26819 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39051/jwks.json
I20260923 17:35:10.550500 26819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:42041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:42041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39051/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:10.552026 26819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:10.553836 26819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:10.565737 26825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:10.565828 26824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:10.565753 26827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:10.566453 26819 server_base.cc:1034] running on GCE node
I20260923 17:35:10.567267 26819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:10.568799 26819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:10.570027 26819 hybrid_clock.cc:648] HybridClock initialized: now 1790184910569935 us; error 89 us; skew 500 ppm
I20260923 17:35:10.570398 26819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:35:10.573002 26819 webserver.cc:460] Webserver started at http://127.20.240.126:38549/ using document root <none> and password file <none>
I20260923 17:35:10.573752 26819 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:10.573890 26819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:10.574224 26819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:10.577265 26819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "ba6ba478ca604aeeb02e5677f999feef"
format_stamp: "Formatted at 2026-09-23 17:35:10 on dist-test-slave-ppjl"
server_key: "73c2bf2c135bf690f49666635e572229"
server_key_iv: "efac602e91eddb9e47f36a3dc941d5e6"
server_key_version: "encryptionkey@0"
I20260923 17:35:10.578101 26819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "ba6ba478ca604aeeb02e5677f999feef"
format_stamp: "Formatted at 2026-09-23 17:35:10 on dist-test-slave-ppjl"
server_key: "73c2bf2c135bf690f49666635e572229"
server_key_iv: "efac602e91eddb9e47f36a3dc941d5e6"
server_key_version: "encryptionkey@0"
I20260923 17:35:10.583614 26819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:35:10.587105 26833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:10.588588 26819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:35:10.588776 26819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "ba6ba478ca604aeeb02e5677f999feef"
format_stamp: "Formatted at 2026-09-23 17:35:10 on dist-test-slave-ppjl"
server_key: "73c2bf2c135bf690f49666635e572229"
server_key_iv: "efac602e91eddb9e47f36a3dc941d5e6"
server_key_version: "encryptionkey@0"
I20260923 17:35:10.588984 26819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:10.608048 26819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:10.608916 26819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:10.609210 26819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:10.626627 26819 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:42041
I20260923 17:35:10.626650 26884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:42041 every 8 connection(s)
I20260923 17:35:10.628098 26819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:35:10.631870 26885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:10.633523 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 26819
I20260923 17:35:10.633700 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:35:10.634323 21441 external_mini_cluster.cc:1428] Setting key 59e895063971dcbadebc4c49747d0803
I20260923 17:35:10.645049 26885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef: Bootstrap starting.
I20260923 17:35:10.649168 26885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:10.650326 26888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:10.636221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54704 (local address 127.20.240.126:42041)
0923 17:35:10.637315 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:10.637330 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:10.637381 (+ 51us) server_negotiation.cc:407] Connection header received
0923 17:35:10.639650 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:10.639680 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:10.640305 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:10.640797 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:10.641383 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:10.643455 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:10.644858 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:10.645894 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:10.646080 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:10.647272 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:10.647285 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:10.647309 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:10.648057 (+ 748us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:10.648203 (+ 146us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:10.648235 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:10.648368 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:10.648600 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 17:35:10.648954 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":303,"threads_started":1}
I20260923 17:35:10.650563 26885 log.cc:824] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:10.653985 26885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef: No bootstrap required, opened a new log
I20260923 17:35:10.654800 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:35:10.661737 26885 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba6ba478ca604aeeb02e5677f999feef" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42041 } }
I20260923 17:35:10.662089 26885 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:10.662186 26885 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba6ba478ca604aeeb02e5677f999feef, State: Initialized, Role: FOLLOWER
I20260923 17:35:10.662761 26885 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba6ba478ca604aeeb02e5677f999feef" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42041 } }
I20260923 17:35:10.662961 26885 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:35:10.663086 26885 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:35:10.663246 26885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:10.665498 26885 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba6ba478ca604aeeb02e5677f999feef" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42041 } }
I20260923 17:35:10.665972 26885 leader_election.cc:302] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba6ba478ca604aeeb02e5677f999feef; no voters:
I20260923 17:35:10.666553 26885 leader_election.cc:288] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:35:10.666728 26890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:10.667454 26890 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [term 1 LEADER]: Becoming Leader. State: Replica: ba6ba478ca604aeeb02e5677f999feef, State: Running, Role: LEADER
I20260923 17:35:10.668004 26890 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba6ba478ca604aeeb02e5677f999feef" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42041 } }
I20260923 17:35:10.668555 26885 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:35:10.670713 26892 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba6ba478ca604aeeb02e5677f999feef. Latest consensus state: current_term: 1 leader_uuid: "ba6ba478ca604aeeb02e5677f999feef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba6ba478ca604aeeb02e5677f999feef" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42041 } } }
I20260923 17:35:10.670881 26891 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba6ba478ca604aeeb02e5677f999feef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba6ba478ca604aeeb02e5677f999feef" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42041 } } }
I20260923 17:35:10.671135 26892 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:10.671255 26891 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:10.672080 26896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:35:10.677942 26896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:35:10.687845 26896 catalog_manager.cc:1347] Generated new cluster ID: df7b27a1d9ab4e51b4e4c84b1bc27aed
I20260923 17:35:10.687975 26896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:35:10.701195 26896 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:35:10.703137 26896 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:35:10.715297 26896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba6ba478ca604aeeb02e5677f999feef: Generated new TSK 0
I20260923 17:35:10.716457 26896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:35:10.743805 26888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:10.735342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.20.240.126:42041)
0923 17:35:10.735570 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:10.735577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:10.736060 (+ 483us) server_negotiation.cc:407] Connection header received
0923 17:35:10.736279 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:10.736285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:10.736387 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:10.736573 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:35:10.736976 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:10.738101 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:10.739478 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:10.740137 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:10.740277 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:10.740836 (+ 559us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:35:10.743121 (+ 2285us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39051/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 17:35:10.743487 (+ 366us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54720: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39051/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":67}
W20260923 17:35:10.743935 26888 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54720: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39051/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 17:35:10.745639 26914 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:10.735348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:42041 (local address 127.0.0.1:54720)
0923 17:35:10.735959 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:10.735976 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:10.736101 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:10.736565 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:10.736569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:10.736588 (+ 19us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:35:10.736831 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:10.736840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:10.738241 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:10.738245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:10.739347 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:10.739360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:10.740367 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:10.740373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:10.740517 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:10.740553 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:35:10.743363 (+ 2810us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39051/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 17:35:10.745210 (+ 1847us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.240.126:42041: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39051/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":1536,"thread_start_us":332,"threads_started":1}
I20260923 17:35:10.746613 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 26819
I20260923 17:35:10.747855 26880 generic_service.cc:196] Checking for leaks (request via RPC)
E20260923 17:35:10.755529 26915 webserver.cc:533] Webserver: sslize failed:
E20260923 17:35:10.755646 26915 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 17:35:10.838666 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 26819
2026-09-23T17:35:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (715 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T17:35:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:35:10Z Disabled control of system clock
I20260923 17:35:10.914671 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:35777
--webserver_interface=127.20.240.126
--webserver_port=0
--builtin_ntp_servers=127.20.240.84:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:35777
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/testchainca.cert with env {}
W20260923 17:35:11.242509 26920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:11.242827 26920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:11.242954 26920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:11.250337 26920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:35:11.250447 26920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:11.250487 26920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:11.250571 26920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:35:11.250638 26920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:35:11.250720 26920 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:35:11.250774 26920 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 17:35:11.260799 26920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:35777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:35777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:11.262426 26920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:11.263839 26920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:11.275923 26925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:11.275979 26926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:11.276481 26928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:11.276692 26920 server_base.cc:1034] running on GCE node
I20260923 17:35:11.277493 26920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:11.278959 26920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:11.280221 26920 hybrid_clock.cc:648] HybridClock initialized: now 1790184911280228 us; error 116 us; skew 500 ppm
I20260923 17:35:11.280560 26920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:35:11.282853 26920 webserver.cc:460] Webserver started at http://127.20.240.126:41573/ using document root <none> and password file <none>
I20260923 17:35:11.283598 26920 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:11.283690 26920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:11.283946 26920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:11.286765 26920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "b5ab193e27294584a9a2836870cb994f"
format_stamp: "Formatted at 2026-09-23 17:35:11 on dist-test-slave-ppjl"
server_key: "a76f93ef142d853d2e383beb12576edb"
server_key_iv: "ce9c81640b67c77b52ade19e91c8b7cc"
server_key_version: "encryptionkey@0"
I20260923 17:35:11.287561 26920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "b5ab193e27294584a9a2836870cb994f"
format_stamp: "Formatted at 2026-09-23 17:35:11 on dist-test-slave-ppjl"
server_key: "a76f93ef142d853d2e383beb12576edb"
server_key_iv: "ce9c81640b67c77b52ade19e91c8b7cc"
server_key_version: "encryptionkey@0"
I20260923 17:35:11.292775 26920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:35:11.296331 26934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:11.297698 26920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:35:11.297840 26920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "b5ab193e27294584a9a2836870cb994f"
format_stamp: "Formatted at 2026-09-23 17:35:11 on dist-test-slave-ppjl"
server_key: "a76f93ef142d853d2e383beb12576edb"
server_key_iv: "ce9c81640b67c77b52ade19e91c8b7cc"
server_key_version: "encryptionkey@0"
I20260923 17:35:11.298025 26920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:11.310783 26920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:11.311648 26920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:11.311887 26920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:11.328873 26920 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:35777
I20260923 17:35:11.328892 26985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:35777 every 8 connection(s)
I20260923 17:35:11.330243 26920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:35:11.333775 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 26920
I20260923 17:35:11.333943 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:35:11.334087 26986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:11.334612 21441 external_mini_cluster.cc:1428] Setting key 8d45b9c53e07af17041211c1387d44f1
I20260923 17:35:11.347333 26986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f: Bootstrap starting.
I20260923 17:35:11.348829 26989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:11.336429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52128 (local address 127.20.240.126:35777)
0923 17:35:11.337427 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:11.337437 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:11.337478 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:35:11.339793 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:11.339816 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:11.340321 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:11.340664 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:11.341255 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:11.342502 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:11.344340 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:11.345059 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:11.345201 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:11.346440 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:11.346448 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:11.346462 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:11.347005 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:11.347091 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:11.347120 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:11.347253 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:11.347461 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 17:35:11.347811 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":335,"threads_started":1}
I20260923 17:35:11.350600 26986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:11.351830 26986 log.cc:824] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:11.352877 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:35:11.354882 26986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f: No bootstrap required, opened a new log
I20260923 17:35:11.362986 26986 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab193e27294584a9a2836870cb994f" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35777 } }
I20260923 17:35:11.363415 26986 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:11.363494 26986 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5ab193e27294584a9a2836870cb994f, State: Initialized, Role: FOLLOWER
I20260923 17:35:11.364001 26986 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab193e27294584a9a2836870cb994f" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35777 } }
I20260923 17:35:11.364257 26986 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:35:11.364414 26986 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:35:11.364576 26986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:11.366705 26986 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab193e27294584a9a2836870cb994f" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35777 } }
I20260923 17:35:11.367244 26986 leader_election.cc:302] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5ab193e27294584a9a2836870cb994f; no voters:
I20260923 17:35:11.367832 26986 leader_election.cc:288] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:35:11.367977 26991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:11.368618 26991 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [term 1 LEADER]: Becoming Leader. State: Replica: b5ab193e27294584a9a2836870cb994f, State: Running, Role: LEADER
I20260923 17:35:11.369172 26991 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab193e27294584a9a2836870cb994f" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35777 } }
I20260923 17:35:11.369843 26986 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:35:11.372349 26992 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5ab193e27294584a9a2836870cb994f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab193e27294584a9a2836870cb994f" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35777 } } }
I20260923 17:35:11.372768 26992 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:11.372613 26993 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5ab193e27294584a9a2836870cb994f. Latest consensus state: current_term: 1 leader_uuid: "b5ab193e27294584a9a2836870cb994f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ab193e27294584a9a2836870cb994f" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35777 } } }
I20260923 17:35:11.372886 26993 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:11.374552 26999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:35:11.380182 26999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:35:11.389889 26999 catalog_manager.cc:1347] Generated new cluster ID: 97dc232a229248df8f37d758aa8ee924
I20260923 17:35:11.390020 26999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:35:11.401429 26999 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:35:11.403417 26999 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:35:11.416174 26999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5ab193e27294584a9a2836870cb994f: Generated new TSK 0
I20260923 17:35:11.417349 26999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:35:12.022099 26989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:11.439700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.20.240.126:35777)
0923 17:35:11.439923 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:11.439930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:11.440352 (+ 422us) server_negotiation.cc:407] Connection header received
0923 17:35:11.440613 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:11.440617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:11.440689 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:11.440842 (+ 153us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:35:11.441376 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:11.442150 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:11.444254 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:11.444703 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:11.444825 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:11.445384 (+ 559us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:35:12.021518 (+576134us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 17:35:12.021875 (+ 357us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52130: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 17:35:12.022214 26989 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52130: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 17:35:12.022409 27015 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:11.439644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35777 (local address 127.0.0.1:52130)
0923 17:35:11.440240 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:11.440265 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:11.440420 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:11.440886 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:11.440892 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:11.440906 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:35:11.441203 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:11.441217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:11.442286 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:11.442291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:11.444083 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:11.444101 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:11.444883 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:11.444889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:11.445071 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:11.445120 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:35:12.021815 (+576695us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 17:35:12.022088 (+ 273us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.240.126:35777: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":283,"threads_started":1}
I20260923 17:35:12.023360 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 26920
I20260923 17:35:12.024952 26981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:12.128985 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 26920
2026-09-23T17:35:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1273 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (402 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (400 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:35:13 dist-test-slave-ppjl krb5kdc[27030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:35:13 dist-test-slave-ppjl krb5kdc[27030](info): set up 2 sockets
Sep 23 17:35:13 dist-test-slave-ppjl krb5kdc[27030](info): commencing operation
krb5kdc: starting...
W20260923 17:35:13.932442 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:35:14 dist-test-slave-ppjl krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:35:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:35:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:35:14.134361 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:33635
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:33635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:14.465813 27043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:14.466125 27043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:14.466249 27043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:14.473658 27043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:35:14.473783 27043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:14.473830 27043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:14.473883 27043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:35:14.473927 27043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:35:14.484212 27043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:33635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:33635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:14.485585 27043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:14.486882 27043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:14.498997 27048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:14.499066 27049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:14.499109 27051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:14.499864 27043 server_base.cc:1034] running on GCE node
I20260923 17:35:14.500705 27043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:14.502175 27043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:14.503374 27043 hybrid_clock.cc:648] HybridClock initialized: now 1790184914503323 us; error 43 us; skew 500 ppm
I20260923 17:35:14.503714 27043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:14 dist-test-slave-ppjl krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184914, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:14.509403 27043 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:35:14.510828 27043 webserver.cc:460] Webserver started at http://127.20.240.126:46097/ using document root <none> and password file <none>
I20260923 17:35:14.511569 27043 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:14.511698 27043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:14.512009 27043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:14.514976 27043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "bfb172a74839400c8d5e49412135d02f"
format_stamp: "Formatted at 2026-09-23 17:35:14 on dist-test-slave-ppjl"
server_key: "58c06631e793a5978a9dcb7c02f99b7c"
server_key_iv: "4a96b71fe22690763f3bdf5da8e4330c"
server_key_version: "encryptionkey@0"
I20260923 17:35:14.515828 27043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "bfb172a74839400c8d5e49412135d02f"
format_stamp: "Formatted at 2026-09-23 17:35:14 on dist-test-slave-ppjl"
server_key: "58c06631e793a5978a9dcb7c02f99b7c"
server_key_iv: "4a96b71fe22690763f3bdf5da8e4330c"
server_key_version: "encryptionkey@0"
I20260923 17:35:14.521386 27043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:35:14.525117 27058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:14.526532 27043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:35:14.526701 27043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "bfb172a74839400c8d5e49412135d02f"
format_stamp: "Formatted at 2026-09-23 17:35:14 on dist-test-slave-ppjl"
server_key: "58c06631e793a5978a9dcb7c02f99b7c"
server_key_iv: "4a96b71fe22690763f3bdf5da8e4330c"
server_key_version: "encryptionkey@0"
I20260923 17:35:14.526895 27043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:14.549247 27043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:14.553969 27043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:14.554282 27043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:14.573758 27043 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:33635
I20260923 17:35:14.573804 27109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:33635 every 8 connection(s)
I20260923 17:35:14.575078 27043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:35:14.576297 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 27043
I20260923 17:35:14.576493 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:35:14.577296 21441 external_mini_cluster.cc:1428] Setting key 72ea4c1bcdb98fbda0b7e15628d3b156
I20260923 17:35:14.579723 27110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:35:14 dist-test-slave-ppjl krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:35:14.590619 27110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f: Bootstrap starting.
I20260923 17:35:14.593895 27110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:14.594965 27110 log.cc:824] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:14.598433 27110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f: No bootstrap required, opened a new log
I20260923 17:35:14.599670 27113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:14.579365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.20.240.126:33635)
0923 17:35:14.580124 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:14.580140 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:14.580194 (+ 54us) server_negotiation.cc:407] Connection header received
0923 17:35:14.581901 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:14.581925 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:14.582322 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:14.582722 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:14.584185 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:14.585488 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:14.586725 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:14.587463 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:14.587660 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:14.591770 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:14.591779 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:14.591797 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:14.591839 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:14.594816 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:14.595662 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:14.595672 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:14.595685 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:14.595874 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:14.596301 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:14.596306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:14.596309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:14.597113 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:14.597400 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:14.597877 (+ 477us) server_negotiation.cc:299] Negotiation successful
0923 17:35:14.598322 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":322,"threads_started":1}
I20260923 17:35:14.606827 27110 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb172a74839400c8d5e49412135d02f" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33635 } }
I20260923 17:35:14.607244 27110 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:14.607342 27110 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfb172a74839400c8d5e49412135d02f, State: Initialized, Role: FOLLOWER
I20260923 17:35:14.607941 27110 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb172a74839400c8d5e49412135d02f" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33635 } }
I20260923 17:35:14.608166 27110 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:35:14.608299 27110 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:35:14.608462 27110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:14.610873 27110 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb172a74839400c8d5e49412135d02f" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33635 } }
I20260923 17:35:14.611456 27110 leader_election.cc:302] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfb172a74839400c8d5e49412135d02f; no voters:
I20260923 17:35:14.612103 27110 leader_election.cc:288] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:35:14.612284 27115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:14.613190 27115 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [term 1 LEADER]: Becoming Leader. State: Replica: bfb172a74839400c8d5e49412135d02f, State: Running, Role: LEADER
I20260923 17:35:14.613737 27115 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb172a74839400c8d5e49412135d02f" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33635 } }
I20260923 17:35:14.614198 27110 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:35:14.616448 27117 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfb172a74839400c8d5e49412135d02f. Latest consensus state: current_term: 1 leader_uuid: "bfb172a74839400c8d5e49412135d02f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb172a74839400c8d5e49412135d02f" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33635 } } }
I20260923 17:35:14.616853 27117 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:14.617308 27116 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfb172a74839400c8d5e49412135d02f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfb172a74839400c8d5e49412135d02f" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33635 } } }
I20260923 17:35:14.617625 27116 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:14.617852 27124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:35:14.623608 27124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:35:14.632581 27124 catalog_manager.cc:1347] Generated new cluster ID: 190614b066b94a8fac0a3a041eb74067
I20260923 17:35:14.632714 27124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:35:14.669328 27124 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:35:14.670668 27124 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:35:14.678748 27124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfb172a74839400c8d5e49412135d02f: Generated new TSK 0
I20260923 17:35:14.679651 27124 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:35:14.739063 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33635
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:15.073841 27136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:15.074160 27136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:15.074285 27136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:15.081853 27136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:15.081986 27136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:15.082149 27136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:35:15.092358 27136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:15.093734 27136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:15.095191 27136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:15.108264 27141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:15.108201 27142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:15.108230 27144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:15.109207 27136 server_base.cc:1034] running on GCE node
I20260923 17:35:15.109840 27136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:15.110858 27136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:15.112064 27136 hybrid_clock.cc:648] HybridClock initialized: now 1790184915112003 us; error 56 us; skew 500 ppm
I20260923 17:35:15.112411 27136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:15 dist-test-slave-ppjl krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184915, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:15.115796 27136 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:35:15.117552 27136 webserver.cc:460] Webserver started at http://127.20.240.65:33639/ using document root <none> and password file <none>
I20260923 17:35:15.118310 27136 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:15.118441 27136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:15.118778 27136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:15.122149 27136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "b2f00679e9c248d59345209e602778ac"
format_stamp: "Formatted at 2026-09-23 17:35:15 on dist-test-slave-ppjl"
server_key: "2e2416f423dfea507db4120f2f09144e"
server_key_iv: "c2e0e4a32639886f7506bcd6a71d159b"
server_key_version: "encryptionkey@0"
I20260923 17:35:15.122994 27136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "b2f00679e9c248d59345209e602778ac"
format_stamp: "Formatted at 2026-09-23 17:35:15 on dist-test-slave-ppjl"
server_key: "2e2416f423dfea507db4120f2f09144e"
server_key_iv: "c2e0e4a32639886f7506bcd6a71d159b"
server_key_version: "encryptionkey@0"
I20260923 17:35:15.128737 27136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:35:15.132603 27151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:15.134153 27136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:35:15.134334 27136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "b2f00679e9c248d59345209e602778ac"
format_stamp: "Formatted at 2026-09-23 17:35:15 on dist-test-slave-ppjl"
server_key: "2e2416f423dfea507db4120f2f09144e"
server_key_iv: "c2e0e4a32639886f7506bcd6a71d159b"
server_key_version: "encryptionkey@0"
I20260923 17:35:15.134537 27136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:15.149425 27136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:15.153663 27136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:15.153932 27136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:15.154915 27136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:15.156332 27136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:15.156423 27136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:15.156546 27136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:15.156611 27136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:15.188557 27136 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:33949
I20260923 17:35:15.188592 27263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:33949 every 8 connection(s)
I20260923 17:35:15.189858 27136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:35:15.191288 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 27136
I20260923 17:35:15.191475 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:35:15.192157 21441 external_mini_cluster.cc:1428] Setting key 040e3cde09f5c07a579e382505233e64
Sep 23 17:35:15 dist-test-slave-ppjl krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184915, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
I20260923 17:35:15.210752 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:15.192664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:37779 (local address 127.20.240.126:33635)
0923 17:35:15.193144 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:15.193148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:15.194450 (+ 1302us) server_negotiation.cc:407] Connection header received
0923 17:35:15.196301 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:15.196306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:15.196384 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:15.196475 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:15.198037 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:15.198780 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:15.200460 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:15.200897 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:15.201026 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:15.204601 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:15.204607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:15.204610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:15.204648 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:15.207356 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:15.208316 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:15.208322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:15.208325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:15.208497 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:15.208957 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:15.208962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:15.208965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:15.209358 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:15.209512 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:15.210274 (+ 762us) server_negotiation.cc:299] Negotiation successful
0923 17:35:15.210465 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":234,"threads_started":1}
I20260923 17:35:15.211830 27267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:15.193445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:33635 (local address 127.20.240.65:37779)
0923 17:35:15.194299 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:15.194352 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:15.196038 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:15.196732 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:15.196742 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:15.197263 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:15.197857 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:15.197877 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:15.198936 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:15.198940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:15.200300 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:15.200316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:15.201145 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:15.201152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:15.201405 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:15.202072 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:15.202096 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:15.204346 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:15.207533 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:15.207541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:15.207557 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:15.208139 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:15.208626 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:15.208630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:15.208632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:15.208830 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:15.209490 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:15.209497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:15.209888 (+ 391us) client_negotiation.cc:769] Sending connection context
0923 17:35:15.210118 (+ 230us) client_negotiation.cc:240] Negotiation successful
0923 17:35:15.210561 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":301,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
I20260923 17:35:15.213621 27264 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33635
I20260923 17:35:15.214030 27264 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:15.214984 27264 heartbeater.cc:511] Master 127.20.240.126:33635 requested a full tablet report, sending...
I20260923 17:35:15.217520 27075 ts_manager.cc:194] Registered new tserver with Master: b2f00679e9c248d59345209e602778ac (127.20.240.65:33949)
I20260923 17:35:15.219481 27075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:37779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:35:15.238545 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33635
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:15.562815 27270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:15.563122 27270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:15.563274 27270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:15.570674 27270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:15.570783 27270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:15.570902 27270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:35:15.581040 27270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:15.582379 27270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:15.583770 27270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:15.596738 27275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:15.596786 27278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:15.596758 27276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:15.597223 27270 server_base.cc:1034] running on GCE node
I20260923 17:35:15.598002 27270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:15.598933 27270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:15.600121 27270 hybrid_clock.cc:648] HybridClock initialized: now 1790184915600056 us; error 51 us; skew 500 ppm
I20260923 17:35:15.600435 27270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:15 dist-test-slave-ppjl krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184915, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:15.603528 27270 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:35:15.604955 27270 webserver.cc:460] Webserver started at http://127.20.240.66:40961/ using document root <none> and password file <none>
I20260923 17:35:15.605584 27270 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:15.605685 27270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:15.605932 27270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:15.608793 27270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "1a4c14e3ec3e43f89d74ede1a799313e"
format_stamp: "Formatted at 2026-09-23 17:35:15 on dist-test-slave-ppjl"
server_key: "29e0259fc00f6ab63bb33984664d55df"
server_key_iv: "d7f8ef1cc59256f3bb103b6d7018c58a"
server_key_version: "encryptionkey@0"
I20260923 17:35:15.609670 27270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "1a4c14e3ec3e43f89d74ede1a799313e"
format_stamp: "Formatted at 2026-09-23 17:35:15 on dist-test-slave-ppjl"
server_key: "29e0259fc00f6ab63bb33984664d55df"
server_key_iv: "d7f8ef1cc59256f3bb103b6d7018c58a"
server_key_version: "encryptionkey@0"
I20260923 17:35:15.614652 27270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:35:15.617985 27285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:15.619407 27270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:35:15.619553 27270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "1a4c14e3ec3e43f89d74ede1a799313e"
format_stamp: "Formatted at 2026-09-23 17:35:15 on dist-test-slave-ppjl"
server_key: "29e0259fc00f6ab63bb33984664d55df"
server_key_iv: "d7f8ef1cc59256f3bb103b6d7018c58a"
server_key_version: "encryptionkey@0"
I20260923 17:35:15.619741 27270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:15.641916 27270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:15.646157 27270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:15.646441 27270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:15.647517 27270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:15.648893 27270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:15.648974 27270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:15.649094 27270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:15.649153 27270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:15.678617 27270 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:43985
I20260923 17:35:15.678663 27397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:43985 every 8 connection(s)
I20260923 17:35:15.680078 27270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:35:15.688383 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 27270
I20260923 17:35:15.688555 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:35:15.688989 21441 external_mini_cluster.cc:1428] Setting key 03ca0fb5ea25409c119913ae4c677ff5
Sep 23 17:35:15 dist-test-slave-ppjl krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184915, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:35:15.701382 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:15.682691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:33203 (local address 127.20.240.126:33635)
0923 17:35:15.682918 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:15.682923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:15.684148 (+ 1225us) server_negotiation.cc:407] Connection header received
0923 17:35:15.686104 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:15.686110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:15.686200 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:15.686291 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:15.688047 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:15.688832 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:15.690884 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:15.691510 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:15.691650 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:15.695343 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:15.695350 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:15.695355 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:15.695398 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:15.698148 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:15.699071 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:15.699075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:15.699077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:15.699263 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:15.699715 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:15.699719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:15.699721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:15.699981 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:15.700091 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:15.700939 (+ 848us) server_negotiation.cc:299] Negotiation successful
0923 17:35:15.701123 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260923 17:35:15.702814 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:15.683185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:33635 (local address 127.20.240.66:33203)
0923 17:35:15.683963 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:15.684011 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:15.685800 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:15.686566 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:15.686576 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:15.687185 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:15.687837 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:15.687858 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:15.688994 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:15.688998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:15.690733 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:15.690748 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:15.691759 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:15.691764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:15.691982 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:15.692717 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:15.692740 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:15.695044 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:15.698311 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:15.698321 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:15.698340 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:15.698958 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:15.699357 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:15.699361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:15.699365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:15.699617 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:15.700101 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:15.700109 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:15.700553 (+ 444us) client_negotiation.cc:769] Sending connection context
0923 17:35:15.700856 (+ 303us) client_negotiation.cc:240] Negotiation successful
0923 17:35:15.701392 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":302,"threads_started":1}
I20260923 17:35:15.704345 27398 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33635
I20260923 17:35:15.704684 27398 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:15.705550 27398 heartbeater.cc:511] Master 127.20.240.126:33635 requested a full tablet report, sending...
I20260923 17:35:15.707152 27075 ts_manager.cc:194] Registered new tserver with Master: 1a4c14e3ec3e43f89d74ede1a799313e (127.20.240.66:43985)
I20260923 17:35:15.708021 27075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:33203
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:35:15.738469 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33635
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:16.060879 27403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:16.061178 27403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:16.061302 27403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:16.068974 27403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:16.069104 27403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:16.069262 27403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:35:16.079622 27403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:16.080971 27403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:16.082250 27403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:16.095196 27408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:16.095062 27411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:16.095094 27409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:16.096202 27403 server_base.cc:1034] running on GCE node
I20260923 17:35:16.096904 27403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:16.097859 27403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:16.099121 27403 hybrid_clock.cc:648] HybridClock initialized: now 1790184916099078 us; error 39 us; skew 500 ppm
I20260923 17:35:16.099517 27403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184916, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:16.102828 27403 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:35:16.104415 27403 webserver.cc:460] Webserver started at http://127.20.240.67:37479/ using document root <none> and password file <none>
I20260923 17:35:16.105125 27403 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:16.105247 27403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:16.105544 27403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:16.108567 27403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "74fee3e1c2a04a77b9580893a628e762"
format_stamp: "Formatted at 2026-09-23 17:35:16 on dist-test-slave-ppjl"
server_key: "0b635b5e63bbfc58e1bb89abba9f4251"
server_key_iv: "6467fa67e15980de77d098e2a1c98fad"
server_key_version: "encryptionkey@0"
I20260923 17:35:16.109344 27403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "74fee3e1c2a04a77b9580893a628e762"
format_stamp: "Formatted at 2026-09-23 17:35:16 on dist-test-slave-ppjl"
server_key: "0b635b5e63bbfc58e1bb89abba9f4251"
server_key_iv: "6467fa67e15980de77d098e2a1c98fad"
server_key_version: "encryptionkey@0"
I20260923 17:35:16.114814 27403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:35:16.118453 27418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:16.120054 27403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:35:16.120220 27403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "74fee3e1c2a04a77b9580893a628e762"
format_stamp: "Formatted at 2026-09-23 17:35:16 on dist-test-slave-ppjl"
server_key: "0b635b5e63bbfc58e1bb89abba9f4251"
server_key_iv: "6467fa67e15980de77d098e2a1c98fad"
server_key_version: "encryptionkey@0"
I20260923 17:35:16.120412 27403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:16.146627 27403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:16.151192 27403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:16.151481 27403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:16.152434 27403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:16.153751 27403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:16.153831 27403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:16.153946 27403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:16.153998 27403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:16.184828 27403 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:39967
I20260923 17:35:16.184898 27530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:39967 every 8 connection(s)
I20260923 17:35:16.186192 27403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:35:16.188356 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 27403
I20260923 17:35:16.188521 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:35:16.189061 21441 external_mini_cluster.cc:1428] Setting key 214971744991d672cb91a38190b5687b
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184916, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:35:16.207494 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:16.189232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:44755 (local address 127.20.240.126:33635)
0923 17:35:16.189461 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:16.189467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:16.190830 (+ 1363us) server_negotiation.cc:407] Connection header received
0923 17:35:16.193314 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:16.193320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:16.193438 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:16.193543 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:16.195124 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.195924 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.197531 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.197947 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.198065 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:16.201801 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:16.201807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:16.201811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:16.201845 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:16.204606 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:16.205307 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:16.205313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:16.205316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:16.205440 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:16.205833 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:16.205838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:16.205841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:16.206108 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:16.206239 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:16.206922 (+ 683us) server_negotiation.cc:299] Negotiation successful
0923 17:35:16.207136 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 17:35:16.208506 27533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:16.189759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:33635 (local address 127.20.240.67:44755)
0923 17:35:16.190673 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:16.190722 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:16.193033 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:16.193806 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:16.193815 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:16.194361 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:16.194938 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:16.194957 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.196089 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.196093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:16.197373 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:16.197387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.198193 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.198201 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:16.198457 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:16.199249 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:16.199275 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:16.201557 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:16.204739 (+ 3182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:16.204748 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:16.204767 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:16.205193 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:16.205544 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:16.205549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:16.205553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:16.205732 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:16.206240 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:16.206248 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:16.206573 (+ 325us) client_negotiation.cc:769] Sending connection context
0923 17:35:16.206813 (+ 240us) client_negotiation.cc:240] Negotiation successful
0923 17:35:16.207365 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":363,"threads_started":1}
I20260923 17:35:16.209817 27531 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33635
I20260923 17:35:16.210119 27531 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:16.210808 27531 heartbeater.cc:511] Master 127.20.240.126:33635 requested a full tablet report, sending...
I20260923 17:35:16.212216 27075 ts_manager.cc:194] Registered new tserver with Master: 74fee3e1c2a04a77b9580893a628e762 (127.20.240.67:39967)
I20260923 17:35:16.213011 27075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:44755
I20260923 17:35:16.217322 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:35:16.222549 27264 heartbeater.cc:503] Master 127.20.240.126:33635 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/test-admin.keytab.
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:16.244822 21441 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 17:35:16.250357 27538 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:16.247796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.20.240.65:33949)
0923 17:35:16.248425 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:16.248434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:16.248461 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:35:16.248521 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:16.248528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:16.248740 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:16.248865 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:16.249852 (+ 987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34022: BlockingRecv error: recv got EOF from 127.0.0.1:34022 (error 108)
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":345,"threads_started":1}
W20260923 17:35:16.250454 21441 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/krb5cc)
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:16.252133 21441 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:16.253679 21441 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:35:16.269407 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:16.255634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34038 (local address 127.20.240.65:33949)
0923 17:35:16.255839 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:16.255845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:16.256015 (+ 170us) server_negotiation.cc:407] Connection header received
0923 17:35:16.256204 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:16.256208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:16.256269 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:16.256351 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:16.257447 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.258468 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.259810 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.260435 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.260550 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:16.264154 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:16.264168 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:16.264182 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:16.264248 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:16.266912 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:16.267600 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:16.267608 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:16.267610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:16.267735 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:16.268014 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:16.268017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:16.268019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:16.268391 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:16.268614 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:16.268876 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 17:35:16.269101 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260923 17:35:16.275424 27538 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:16.273535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34050 (local address 127.20.240.65:33949)
0923 17:35:16.273716 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:16.273723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:16.274025 (+ 302us) server_negotiation.cc:407] Connection header received
0923 17:35:16.274182 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:16.274186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:16.274266 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:16.274345 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:16.275121 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34050: BlockingRecv error: recv got EOF from 127.0.0.1:34050 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260923 17:35:16.275904 21441 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/krb5cc)
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:16.277412 21441 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:16.278846 21441 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:35:16.292703 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:16.280216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.20.240.65:33949)
0923 17:35:16.280362 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:16.280366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:16.280672 (+ 306us) server_negotiation.cc:407] Connection header received
0923 17:35:16.280958 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:16.280963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:16.281097 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:16.281188 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:16.282077 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.282801 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.284122 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.284542 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.284645 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:16.288056 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:16.288062 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:16.288065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:16.288113 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:16.290515 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:16.291108 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:16.291112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:16.291114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:16.291324 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:16.291639 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:16.291642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:16.291644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:16.291895 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:16.292089 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:16.292318 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 17:35:16.292482 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260923 17:35:16.296759 27538 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:16.295282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34064 (local address 127.20.240.65:33949)
0923 17:35:16.295471 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:16.295477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:16.295614 (+ 137us) server_negotiation.cc:407] Connection header received
0923 17:35:16.295770 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:16.295774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:16.295845 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:16.295920 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:16.296547 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34064: BlockingRecv error: recv got EOF from 127.0.0.1:34064 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260923 17:35:16.297374 21441 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184865977475-21441-0/krb5kdc/krb5cc)
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:16.298815 21441 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:16.300050 21441 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:35:16.314155 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:16.301402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34076 (local address 127.20.240.65:33949)
0923 17:35:16.301592 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:16.301598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:16.301862 (+ 264us) server_negotiation.cc:407] Connection header received
0923 17:35:16.302021 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:16.302025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:16.302085 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:16.302212 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:16.303169 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.303897 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.305086 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.305510 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.305616 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:16.309080 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:16.309086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:16.309089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:16.309130 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:16.311675 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:16.312292 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:16.312296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:16.312299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:16.312443 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:16.312797 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:16.312800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:16.312802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:16.313092 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:16.313381 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:16.313609 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 17:35:16.313842 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:35:16.315759 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 27136
I20260923 17:35:16.327479 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:16.316542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.20.240.65:33949)
0923 17:35:16.316718 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:16.316724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:16.316960 (+ 236us) server_negotiation.cc:407] Connection header received
0923 17:35:16.317141 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:16.317146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:16.317201 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:16.317280 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:16.318172 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.318836 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.320095 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.320514 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.320619 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:16.322505 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:16.322510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:16.322513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:16.322541 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:16.325288 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:16.325914 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:16.325918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:16.325921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:16.326055 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:16.326363 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:16.326366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:16.326368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:16.326613 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:16.326798 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:16.327014 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 17:35:16.327213 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:35:16.328059 27259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:16.411419 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 27136
I20260923 17:35:16.424530 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 27270
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:35:16.440770 27551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:16.425338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.20.240.66:43985)
0923 17:35:16.425887 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:16.425894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:16.425919 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:35:16.425979 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:16.425987 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:16.426222 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:16.426354 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:16.427473 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.428557 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.430001 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.430572 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.430688 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:16.434491 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:16.434505 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:16.434520 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:16.434568 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:16.437602 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:16.438360 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:16.438368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:16.438370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:16.438507 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:16.438912 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:16.438916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:16.438918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:16.439503 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:16.439752 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:16.440181 (+ 429us) server_negotiation.cc:299] Negotiation successful
0923 17:35:16.440449 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":269,"threads_started":1}
I20260923 17:35:16.442062 27393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:16.525033 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 27270
I20260923 17:35:16.537828 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 27403
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27030](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:35:16.554970 27553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:16.538963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.20.240.67:39967)
0923 17:35:16.539625 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:16.539632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:16.539661 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:35:16.539756 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:16.539763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:16.539981 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:16.540108 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:16.541319 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.542358 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.543835 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:16.544441 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:16.544547 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:16.548733 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:16.548751 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:16.548768 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:16.548822 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:16.551859 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:16.552757 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:16.552766 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:16.552768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:16.552916 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:16.553345 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:16.553350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:16.553352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:16.553740 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:16.553999 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:16.554361 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 17:35:16.554650 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":361,"threads_started":1}
I20260923 17:35:16.556608 27526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:16.647464 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 27403
I20260923 17:35:16.660090 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 27043
I20260923 17:35:16.661432 27105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:16.766004 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 27043
2026-09-23T17:35:16Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3836 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27558](info): set up 2 sockets
Sep 23 17:35:16 dist-test-slave-ppjl krb5kdc[27558](info): commencing operation
krb5kdc: starting...
W20260923 17:35:17.768405 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:35:17 dist-test-slave-ppjl krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:35:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:35:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.126.keytab.
Entry for principal oryx/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.126.keytab.
I20260923 17:35:17.974864 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:36193
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.126.keytab
--principal=oryx/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:36193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:18.314198 27571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:18.314493 27571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:18.314584 27571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:18.321866 27571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:35:18.321996 27571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:18.322050 27571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:18.322106 27571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:35:18.322149 27571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:35:18.332265 27571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:36193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.126.keytab
--principal=oryx/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:36193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:18.333647 27571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:18.334985 27571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:18.346999 27576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:18.347267 27577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:18.347301 27579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:18.347990 27571 server_base.cc:1034] running on GCE node
I20260923 17:35:18.348858 27571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:18.350466 27571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:18.351686 27571 hybrid_clock.cc:648] HybridClock initialized: now 1790184918351623 us; error 53 us; skew 500 ppm
I20260923 17:35:18.352065 27571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:18 dist-test-slave-ppjl krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184918, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:18.355468 27571 init.cc:377] Logged in from keytab as oryx/127.20.240.126@KRBTEST.COM (short username oryx)
I20260923 17:35:18.357061 27571 webserver.cc:460] Webserver started at http://127.20.240.126:33243/ using document root <none> and password file <none>
I20260923 17:35:18.357785 27571 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:18.357901 27571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:18.358227 27571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:18.361469 27571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "9dc5d7a24a374aefa3e191059db69da4"
format_stamp: "Formatted at 2026-09-23 17:35:18 on dist-test-slave-ppjl"
server_key: "85a864d4046232156dc7e3673e8c715a"
server_key_iv: "9b81d271a3e7bee8054a354cfbab8230"
server_key_version: "encryptionkey@0"
I20260923 17:35:18.362274 27571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "9dc5d7a24a374aefa3e191059db69da4"
format_stamp: "Formatted at 2026-09-23 17:35:18 on dist-test-slave-ppjl"
server_key: "85a864d4046232156dc7e3673e8c715a"
server_key_iv: "9b81d271a3e7bee8054a354cfbab8230"
server_key_version: "encryptionkey@0"
I20260923 17:35:18.367841 27571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:35:18.371479 27586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:18.373024 27571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:35:18.373225 27571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "9dc5d7a24a374aefa3e191059db69da4"
format_stamp: "Formatted at 2026-09-23 17:35:18 on dist-test-slave-ppjl"
server_key: "85a864d4046232156dc7e3673e8c715a"
server_key_iv: "9b81d271a3e7bee8054a354cfbab8230"
server_key_version: "encryptionkey@0"
I20260923 17:35:18.373430 27571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:18.397148 27571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:18.401829 27571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:18.402124 27571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:18.419678 27571 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:36193
I20260923 17:35:18.419721 27637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:36193 every 8 connection(s)
I20260923 17:35:18.421093 27571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:35:18.425050 27638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:18.425205 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 27571
I20260923 17:35:18.425410 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:35:18.426138 21441 external_mini_cluster.cc:1428] Setting key af824efe2e48183f47edc94d14a65b70
I20260923 17:35:18.440049 27638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4: Bootstrap starting.
Sep 23 17:35:18 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
I20260923 17:35:18.443470 27638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:18.444852 27638 log.cc:824] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:18.448343 27638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4: No bootstrap required, opened a new log
I20260923 17:35:18.451560 27641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:18.428301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.20.240.126:36193)
0923 17:35:18.429139 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:18.429165 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:18.429217 (+ 52us) server_negotiation.cc:407] Connection header received
0923 17:35:18.431744 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:18.431780 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:18.432429 (+ 649us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:18.432944 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:18.434403 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:18.436503 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:18.437887 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:18.438990 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:18.439206 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:18.443140 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:18.443188 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:18.443210 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:18.443268 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:18.446794 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:18.447608 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:18.447617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:18.447626 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:18.447843 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:18.448186 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:18.448191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:18.448194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:18.448926 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:18.449257 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:18.449616 (+ 359us) server_negotiation.cc:299] Negotiation successful
0923 17:35:18.450068 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":342,"threads_started":1}
I20260923 17:35:18.457823 27638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } }
I20260923 17:35:18.458238 27638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:18.458312 27638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc5d7a24a374aefa3e191059db69da4, State: Initialized, Role: FOLLOWER
I20260923 17:35:18.459033 27638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } }
I20260923 17:35:18.459367 27638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:35:18.459528 27638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:35:18.459740 27638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:18.461959 27638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } }
I20260923 17:35:18.462438 27638 leader_election.cc:302] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dc5d7a24a374aefa3e191059db69da4; no voters:
I20260923 17:35:18.463037 27638 leader_election.cc:288] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:35:18.463178 27643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:18.463795 27643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [term 1 LEADER]: Becoming Leader. State: Replica: 9dc5d7a24a374aefa3e191059db69da4, State: Running, Role: LEADER
I20260923 17:35:18.464361 27643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } }
I20260923 17:35:18.465227 27638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:35:18.467379 27644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } }
I20260923 17:35:18.467782 27644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:18.468437 27645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dc5d7a24a374aefa3e191059db69da4. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } }
I20260923 17:35:18.468868 27645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:18.469467 27652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:35:18.473940 27652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:35:18.483737 27652 catalog_manager.cc:1347] Generated new cluster ID: f902ec2a20da4b6f917072be6129bc21
I20260923 17:35:18.483870 27652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:35:18.500192 27652 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:35:18.501605 27652 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:35:18.512436 27652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4: Generated new TSK 0
I20260923 17:35:18.513389 27652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.65.keytab.
Entry for principal oryx/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.65.keytab.
I20260923 17:35:18.574390 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:36193
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.65.keytab
--principal=oryx/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:18.921811 27664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:18.922142 27664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:18.922237 27664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:18.929818 27664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:18.929960 27664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:18.930091 27664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:35:18.940755 27664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.65.keytab
--principal=oryx/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:36193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:18.942122 27664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:18.943560 27664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:18.956938 27669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:18.957003 27672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:18.956969 27670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:18.957726 27664 server_base.cc:1034] running on GCE node
I20260923 17:35:18.958369 27664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:18.959312 27664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:18.960542 27664 hybrid_clock.cc:648] HybridClock initialized: now 1790184918960481 us; error 52 us; skew 500 ppm
I20260923 17:35:18.960865 27664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:18 dist-test-slave-ppjl krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184918, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:18.964026 27664 init.cc:377] Logged in from keytab as oryx/127.20.240.65@KRBTEST.COM (short username oryx)
I20260923 17:35:18.965472 27664 webserver.cc:460] Webserver started at http://127.20.240.65:34621/ using document root <none> and password file <none>
I20260923 17:35:18.966172 27664 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:18.966290 27664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:18.966566 27664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:18.969686 27664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "4dbaafbf88034b6da6cfd705d4849bba"
format_stamp: "Formatted at 2026-09-23 17:35:18 on dist-test-slave-ppjl"
server_key: "3ab5bb85d6a25fbd8ddcd394a4d72e98"
server_key_iv: "e6cbb8563101f9ca30f8528440d69711"
server_key_version: "encryptionkey@0"
I20260923 17:35:18.970472 27664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "4dbaafbf88034b6da6cfd705d4849bba"
format_stamp: "Formatted at 2026-09-23 17:35:18 on dist-test-slave-ppjl"
server_key: "3ab5bb85d6a25fbd8ddcd394a4d72e98"
server_key_iv: "e6cbb8563101f9ca30f8528440d69711"
server_key_version: "encryptionkey@0"
I20260923 17:35:18.975772 27664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:35:18.979365 27679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:18.980854 27664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:35:18.981015 27664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "4dbaafbf88034b6da6cfd705d4849bba"
format_stamp: "Formatted at 2026-09-23 17:35:18 on dist-test-slave-ppjl"
server_key: "3ab5bb85d6a25fbd8ddcd394a4d72e98"
server_key_iv: "e6cbb8563101f9ca30f8528440d69711"
server_key_version: "encryptionkey@0"
I20260923 17:35:18.981212 27664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:19.000813 27664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:19.005165 27664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:19.005447 27664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:19.006451 27664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:19.007963 27664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:19.008051 27664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:19.008184 27664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:19.008244 27664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:19.038666 27664 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:37703
I20260923 17:35:19.038712 27791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:37703 every 8 connection(s)
I20260923 17:35:19.039994 27664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:35:19.048499 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 27664
I20260923 17:35:19.048714 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:35:19.049328 21441 external_mini_cluster.cc:1428] Setting key 109f91affc887597a7f6f9be8efd04b2
Sep 23 17:35:19 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184918, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
I20260923 17:35:19.061307 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:19.042641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:35081 (local address 127.20.240.126:36193)
0923 17:35:19.043095 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:19.043100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:19.044056 (+ 956us) server_negotiation.cc:407] Connection header received
0923 17:35:19.045877 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:19.045883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:19.045971 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:19.046067 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:19.047682 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:19.048458 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:19.050224 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:19.050795 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:19.050932 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:19.055100 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:19.055106 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:19.055109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:19.055149 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:19.057841 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:19.058735 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:19.058740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:19.058742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:19.058883 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:19.059415 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:19.059419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:19.059421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:19.059723 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:19.059871 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:19.060643 (+ 772us) server_negotiation.cc:299] Negotiation successful
0923 17:35:19.060932 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":346,"thread_start_us":243,"threads_started":1}
I20260923 17:35:19.062350 27795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:19.043129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.65:35081)
0923 17:35:19.043899 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:19.043936 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:19.045598 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:19.046328 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:19.046336 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:19.046849 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:19.047505 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:19.047524 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:19.048601 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:19.048605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:19.049997 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:19.050012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:19.051076 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:19.051085 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:19.051417 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:19.052206 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:19.052238 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:19.054801 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:19.058043 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:19.058052 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:19.058068 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:19.058568 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:19.059019 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:19.059022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:19.059025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:19.059291 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:19.059856 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:19.059865 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:19.060259 (+ 394us) client_negotiation.cc:769] Sending connection context
0923 17:35:19.060521 (+ 262us) client_negotiation.cc:240] Negotiation successful
0923 17:35:19.061117 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":266,"threads_started":1}
I20260923 17:35:19.063999 27792 heartbeater.cc:347] Connected to a master server at 127.20.240.126:36193
I20260923 17:35:19.064309 27792 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:19.065125 27792 heartbeater.cc:511] Master 127.20.240.126:36193 requested a full tablet report, sending...
I20260923 17:35:19.067656 27603 ts_manager.cc:194] Registered new tserver with Master: 4dbaafbf88034b6da6cfd705d4849bba (127.20.240.65:37703)
WARNING: no policy specified for oryx/127.20.240.66@KRBTEST.COM; defaulting to no policy
I20260923 17:35:19.069366 27603 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:35081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.66.keytab.
Entry for principal oryx/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.66.keytab.
I20260923 17:35:19.096426 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:36193
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.66.keytab
--principal=oryx/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:19.417840 27798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:19.418185 27798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:19.418290 27798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:19.425797 27798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:19.425910 27798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:19.426167 27798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:35:19.436501 27798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.66.keytab
--principal=oryx/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:36193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:19.437732 27798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:19.438994 27798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:19.451983 27803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:19.451946 27804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:19.451910 27806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:19.452605 27798 server_base.cc:1034] running on GCE node
I20260923 17:35:19.453174 27798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:19.454249 27798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:19.455451 27798 hybrid_clock.cc:648] HybridClock initialized: now 1790184919455327 us; error 89 us; skew 500 ppm
I20260923 17:35:19.455845 27798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:19 dist-test-slave-ppjl krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184919, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:19.459853 27798 init.cc:377] Logged in from keytab as oryx/127.20.240.66@KRBTEST.COM (short username oryx)
I20260923 17:35:19.461453 27798 webserver.cc:460] Webserver started at http://127.20.240.66:41503/ using document root <none> and password file <none>
I20260923 17:35:19.462327 27798 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:19.462460 27798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:19.462752 27798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:19.465816 27798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "841cba3f424b4c6ea31bb73a14f17314"
format_stamp: "Formatted at 2026-09-23 17:35:19 on dist-test-slave-ppjl"
server_key: "1463d2a2607e8d3ab27094f57de4b608"
server_key_iv: "7d40f4aa79082310045090b0c668244f"
server_key_version: "encryptionkey@0"
I20260923 17:35:19.466614 27798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "841cba3f424b4c6ea31bb73a14f17314"
format_stamp: "Formatted at 2026-09-23 17:35:19 on dist-test-slave-ppjl"
server_key: "1463d2a2607e8d3ab27094f57de4b608"
server_key_iv: "7d40f4aa79082310045090b0c668244f"
server_key_version: "encryptionkey@0"
I20260923 17:35:19.471611 27798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:35:19.475215 27813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:19.476649 27798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:35:19.476819 27798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "841cba3f424b4c6ea31bb73a14f17314"
format_stamp: "Formatted at 2026-09-23 17:35:19 on dist-test-slave-ppjl"
server_key: "1463d2a2607e8d3ab27094f57de4b608"
server_key_iv: "7d40f4aa79082310045090b0c668244f"
server_key_version: "encryptionkey@0"
I20260923 17:35:19.476992 27798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:19.497258 27798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:19.501680 27798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:19.502002 27798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:19.503021 27798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:19.504421 27798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:19.504493 27798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:19.504603 27798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:19.504654 27798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:19.534228 27798 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:35803
I20260923 17:35:19.534263 27925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:35803 every 8 connection(s)
I20260923 17:35:19.535527 27798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:35:19.536190 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 27798
I20260923 17:35:19.536372 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:35:19.536886 21441 external_mini_cluster.cc:1428] Setting key 3e49f8884a54a710985abedf57ce9c22
Sep 23 17:35:19 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184919, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.20.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.240.67@KRBTEST.COM" created.
I20260923 17:35:19.562304 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:19.538146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:54433 (local address 127.20.240.126:36193)
0923 17:35:19.538330 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:19.538334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:19.539767 (+ 1433us) server_negotiation.cc:407] Connection header received
0923 17:35:19.542011 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:19.542016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:19.542102 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:19.542196 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:19.543866 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:19.544853 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:19.546415 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:19.547057 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:19.547206 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:19.556263 (+ 9057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:19.556269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:19.556272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:19.556314 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:19.559063 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:19.559918 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:19.559922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:19.559924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:19.560055 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:19.560510 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:19.560514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:19.560517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:19.560817 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:19.560951 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:19.561703 (+ 752us) server_negotiation.cc:299] Negotiation successful
0923 17:35:19.561949 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:35:19.563408 27928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:19.538851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.66:54433)
0923 17:35:19.539607 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:19.539651 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:19.541741 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:19.542443 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:19.542452 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:19.542979 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:19.543658 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:19.543697 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:19.545013 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:19.545017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:19.546263 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:19.546275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:19.547296 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:19.547301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:19.547505 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:19.553356 (+ 5851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:19.553392 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:19.555958 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:19.559293 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:19.559302 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:19.559318 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:19.559772 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:19.560180 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:19.560185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:19.560189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:19.560387 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:19.560945 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:19.560954 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:19.561327 (+ 373us) client_negotiation.cc:769] Sending connection context
0923 17:35:19.561571 (+ 244us) client_negotiation.cc:240] Negotiation successful
0923 17:35:19.562160 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":327,"threads_started":1}
I20260923 17:35:19.564814 27926 heartbeater.cc:347] Connected to a master server at 127.20.240.126:36193
I20260923 17:35:19.565131 27926 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:19.565944 27926 heartbeater.cc:511] Master 127.20.240.126:36193 requested a full tablet report, sending...
I20260923 17:35:19.567468 27603 ts_manager.cc:194] Registered new tserver with Master: 841cba3f424b4c6ea31bb73a14f17314 (127.20.240.66:35803)
I20260923 17:35:19.568303 27603 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:54433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.67.keytab.
Entry for principal oryx/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.67.keytab.
I20260923 17:35:19.582726 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:36193
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.67.keytab
--principal=oryx/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:19.898579 27931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:19.898867 27931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:19.899012 27931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:19.906445 27931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:19.906581 27931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:19.906741 27931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:35:19.916777 27931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.67.keytab
--principal=oryx/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:36193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:19.918112 27931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:19.919525 27931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:19.932456 27936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:19.932478 27937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:19.932997 27939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:19.933544 27931 server_base.cc:1034] running on GCE node
I20260923 17:35:19.934237 27931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:19.935236 27931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:19.936465 27931 hybrid_clock.cc:648] HybridClock initialized: now 1790184919936397 us; error 64 us; skew 500 ppm
I20260923 17:35:19.936801 27931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:19 dist-test-slave-ppjl krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184919, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:19.940092 27931 init.cc:377] Logged in from keytab as oryx/127.20.240.67@KRBTEST.COM (short username oryx)
I20260923 17:35:19.941771 27931 webserver.cc:460] Webserver started at http://127.20.240.67:33653/ using document root <none> and password file <none>
I20260923 17:35:19.942513 27931 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:19.942641 27931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:19.942940 27931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:19.946030 27931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "494adc561a604d76954288bed8325b03"
format_stamp: "Formatted at 2026-09-23 17:35:19 on dist-test-slave-ppjl"
server_key: "b15b992e2b45d8465a83e3b4f36e8b51"
server_key_iv: "2873f86b768ede6eeaab5be8195e50e3"
server_key_version: "encryptionkey@0"
I20260923 17:35:19.946828 27931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "494adc561a604d76954288bed8325b03"
format_stamp: "Formatted at 2026-09-23 17:35:19 on dist-test-slave-ppjl"
server_key: "b15b992e2b45d8465a83e3b4f36e8b51"
server_key_iv: "2873f86b768ede6eeaab5be8195e50e3"
server_key_version: "encryptionkey@0"
I20260923 17:35:19.952121 27931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:35:19.955790 27946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:19.957273 27931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:35:19.957437 27931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "494adc561a604d76954288bed8325b03"
format_stamp: "Formatted at 2026-09-23 17:35:19 on dist-test-slave-ppjl"
server_key: "b15b992e2b45d8465a83e3b4f36e8b51"
server_key_iv: "2873f86b768ede6eeaab5be8195e50e3"
server_key_version: "encryptionkey@0"
I20260923 17:35:19.957626 27931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:19.988065 27931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:19.992974 27931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:19.993317 27931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:19.994375 27931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:19.995975 27931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:19.996054 27931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:19.996166 27931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:19.996217 27931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:20.024655 27931 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:45969
I20260923 17:35:20.024695 28058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:45969 every 8 connection(s)
I20260923 17:35:20.025911 27931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:35:20.031781 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 27931
I20260923 17:35:20.031997 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:35:20.032548 21441 external_mini_cluster.cc:1428] Setting key 9b71b304016ff26c70a9c99ed944a17b
Sep 23 17:35:20 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184919, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
I20260923 17:35:20.046780 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:20.028752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:35779 (local address 127.20.240.126:36193)
0923 17:35:20.028929 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:20.028934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:20.030023 (+ 1089us) server_negotiation.cc:407] Connection header received
0923 17:35:20.031936 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:20.031943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:20.032044 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:20.032170 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:20.033716 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.034514 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.036127 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.036525 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.036644 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:20.040309 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:20.040314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:20.040317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:20.040353 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:20.043511 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.044300 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.044305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.044308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.044474 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.044914 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.044918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.044920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.045283 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:20.045412 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:20.046168 (+ 756us) server_negotiation.cc:299] Negotiation successful
0923 17:35:20.046409 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:35:20.047780 28061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:20.029149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.67:35779)
0923 17:35:20.029843 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:20.029887 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:20.031623 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:20.032383 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:20.032392 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:20.032937 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:20.033540 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:20.033557 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.034660 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.034664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:20.035990 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:20.036003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.036742 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.036749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:20.036975 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:20.037731 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:20.037755 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:20.040082 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:20.043670 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:20.043677 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:20.043706 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:20.044143 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:20.044580 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:20.044583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:20.044586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:20.044793 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:20.045387 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:20.045392 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:20.045749 (+ 357us) client_negotiation.cc:769] Sending connection context
0923 17:35:20.045986 (+ 237us) client_negotiation.cc:240] Negotiation successful
0923 17:35:20.046431 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":267,"threads_started":1}
I20260923 17:35:20.049238 28059 heartbeater.cc:347] Connected to a master server at 127.20.240.126:36193
I20260923 17:35:20.049546 28059 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:20.050288 28059 heartbeater.cc:511] Master 127.20.240.126:36193 requested a full tablet report, sending...
I20260923 17:35:20.051868 27603 ts_manager.cc:194] Registered new tserver with Master: 494adc561a604d76954288bed8325b03 (127.20.240.67:45969)
I20260923 17:35:20.052686 27603 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:35779
I20260923 17:35:20.061298 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:35:20.073135 27792 heartbeater.cc:503] Master 127.20.240.126:36193 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.20.240.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.240.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.240.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.125.keytab.
Entry for principal oryx/127.20.240.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.125.keytab.
I20260923 17:35:20.111299 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.125:43291
--webserver_interface=127.20.240.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.125.keytab
--principal=oryx/127.20.240.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:36193,127.20.240.125:43291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:20.437414 28064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:20.437727 28064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:20.437860 28064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:20.445346 28064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:35:20.445482 28064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:20.445571 28064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:20.445647 28064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:35:20.445708 28064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:35:20.455811 28064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:36193,127.20.240.125:43291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.125.keytab
--principal=oryx/127.20.240.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.125:43291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.240.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
W20260923 17:35:20.456209 28064 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.240.126:36193,127.20.240.125:43291'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 17:35:20.457268 28064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:20.458607 28064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:20.470361 28070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:20.470376 28072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:20.470381 28069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:20.471279 28064 server_base.cc:1034] running on GCE node
I20260923 17:35:20.472213 28064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:20.473723 28064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:20.474952 28064 hybrid_clock.cc:648] HybridClock initialized: now 1790184920474868 us; error 65 us; skew 500 ppm
I20260923 17:35:20.475337 28064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:20 dist-test-slave-ppjl krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184920, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:20.478591 28064 init.cc:377] Logged in from keytab as oryx/127.20.240.125@KRBTEST.COM (short username oryx)
I20260923 17:35:20.480176 28064 webserver.cc:460] Webserver started at http://127.20.240.125:44085/ using document root <none> and password file <none>
I20260923 17:35:20.480962 28064 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:20.481067 28064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:20.481370 28064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:20.484596 28064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/data/instance:
uuid: "90f6ec2f0d9740be81aa4929afc03075"
format_stamp: "Formatted at 2026-09-23 17:35:20 on dist-test-slave-ppjl"
server_key: "f682dd6551c0ee23e2714d76afb62963"
server_key_iv: "41fc2d0dd8338b19e7b36af0a7706140"
server_key_version: "encryptionkey@0"
I20260923 17:35:20.485389 28064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/wal/instance:
uuid: "90f6ec2f0d9740be81aa4929afc03075"
format_stamp: "Formatted at 2026-09-23 17:35:20 on dist-test-slave-ppjl"
server_key: "f682dd6551c0ee23e2714d76afb62963"
server_key_iv: "41fc2d0dd8338b19e7b36af0a7706140"
server_key_version: "encryptionkey@0"
I20260923 17:35:20.490767 28064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:35:20.494328 28079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:20.495800 28064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:35:20.495963 28064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/wal
uuid: "90f6ec2f0d9740be81aa4929afc03075"
format_stamp: "Formatted at 2026-09-23 17:35:20 on dist-test-slave-ppjl"
server_key: "f682dd6551c0ee23e2714d76afb62963"
server_key_iv: "41fc2d0dd8338b19e7b36af0a7706140"
server_key_version: "encryptionkey@0"
I20260923 17:35:20.496181 28064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:20.518496 28064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:20.522821 28064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:20.523075 28064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:20.541401 28064 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.125:43291
I20260923 17:35:20.541431 28130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.125:43291 every 8 connection(s)
I20260923 17:35:20.542745 28064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/data/info.pb
I20260923 17:35:20.546306 28131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:20.550872 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 28064
I20260923 17:35:20.553330 28131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:35:20 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184920, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.125@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
Sep 23 17:35:20 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.240.125@KRBTEST.COM
I20260923 17:35:20.570823 27926 heartbeater.cc:503] Master 127.20.240.126:36193 was elected leader, sending a full tablet report...
I20260923 17:35:20.572041 28135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:20.556656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.20.240.126:36193)
0923 17:35:20.557113 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:20.557118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:20.557829 (+ 711us) server_negotiation.cc:407] Connection header received
0923 17:35:20.557893 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:20.557896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:20.557965 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:20.558054 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:20.559357 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.560104 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.561611 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.562066 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.562188 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:20.565816 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:20.565824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:20.565829 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:20.565875 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:20.568531 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.569441 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.569445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.569447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.569561 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.569969 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.569974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.569978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.570375 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:20.570495 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:20.571451 (+ 956us) server_negotiation.cc:299] Negotiation successful
0923 17:35:20.571677 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":218,"threads_started":1}
I20260923 17:35:20.573144 28136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:20.556649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:46600)
0923 17:35:20.557494 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:20.557539 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:20.557740 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:20.558153 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:20.558161 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:20.558751 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:20.559191 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:20.559213 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.560260 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.560264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:20.561468 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:20.561488 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.562289 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.562297 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:20.562444 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:20.563144 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:20.563209 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:20.565597 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:20.568680 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:20.568689 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:20.568703 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:20.569336 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:20.569667 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:20.569671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:20.569674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:20.569863 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:20.570489 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:20.570497 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:20.570998 (+ 501us) client_negotiation.cc:769] Sending connection context
0923 17:35:20.571335 (+ 337us) client_negotiation.cc:240] Negotiation successful
0923 17:35:20.571840 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"spinlock_wait_cycles":8960,"thread_start_us":446,"threads_started":1}
I20260923 17:35:20.573196 28134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:20.553292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51296 (local address 127.20.240.125:43291)
0923 17:35:20.554156 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:20.554173 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:20.554226 (+ 53us) server_negotiation.cc:407] Connection header received
0923 17:35:20.556441 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:20.556471 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:20.556949 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:20.557392 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:20.558925 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.560196 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.561464 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.562052 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.562221 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:20.566513 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:20.566524 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:20.566540 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:20.566581 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:20.569593 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.570289 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.570296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.570298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.570403 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.570778 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.570783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.570786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.571223 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:20.571504 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:20.572567 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0923 17:35:20.572815 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"spinlock_wait_cycles":79744,"thread_start_us":352,"threads_started":1}
I20260923 17:35:20.574970 28131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:35:20 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184920, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.125@KRBTEST.COM for oryx/127.20.240.125@KRBTEST.COM
I20260923 17:35:20.586906 28136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:20.575733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:43291 (local address 127.0.0.1:51312)
0923 17:35:20.575883 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:20.575901 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:20.575986 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:20.576390 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:20.576393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:20.576692 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:20.576825 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:20.576833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.577820 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.577859 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:20.578833 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:20.578845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.579511 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.579514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:20.579645 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:20.580159 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:20.580175 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:20.582197 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:20.584802 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:20.584805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:20.584808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:20.585143 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:20.585449 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:20.585452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:20.585454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:20.585611 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:20.586112 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:20.586115 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:20.586231 (+ 116us) client_negotiation.cc:769] Sending connection context
0923 17:35:20.586334 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 17:35:20.586604 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 17:35:20.586977 28134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:20.575711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51312 (local address 127.20.240.125:43291)
0923 17:35:20.576160 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:20.576164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:20.576180 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:35:20.576217 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:20.576220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:20.576271 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:20.576353 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:20.576939 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.577687 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.578960 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.579396 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.579490 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:20.582364 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:20.582369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:20.582372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:20.582400 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:20.584684 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.585239 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.585242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.585244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.585362 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.585730 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.585735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.585738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.586022 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:20.586129 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:20.586408 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 17:35:20.586741 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":65}
I20260923 17:35:20.591589 28131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: Bootstrap starting.
I20260923 17:35:20.594602 28131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:20.595646 28131 log.cc:824] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:20.598115 28131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: No bootstrap required, opened a new log
I20260923 17:35:20.605218 28131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } }
I20260923 17:35:20.605564 28131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:20.605662 28131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f6ec2f0d9740be81aa4929afc03075, State: Initialized, Role: FOLLOWER
I20260923 17:35:20.606225 28131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } }
I20260923 17:35:20.607028 28138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } } }
I20260923 17:35:20.607379 28138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:35:20.608034 28131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:35:20.616436 28149 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:35:20.616595 28149 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:35:20.618932 28064 master_runner.cc:428] Detected that this master 90f6ec2f0d9740be81aa4929afc03075 is joining an existing cluster
I20260923 17:35:20.619012 28064 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.240.125:43291
I20260923 17:35:20.639533 28135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:20.622789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.20.240.126:36193)
0923 17:35:20.622981 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:20.622987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:20.623971 (+ 984us) server_negotiation.cc:407] Connection header received
0923 17:35:20.624142 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:20.624148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:20.624270 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:20.624367 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:20.625886 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.626783 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.628844 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.629409 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.629510 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:20.632109 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:20.632116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:20.632120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:20.632158 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:20.635326 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.637269 (+ 1943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.637275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.637278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.637442 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.637919 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.637924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.637928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.638331 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:20.638488 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:20.638926 (+ 438us) server_negotiation.cc:299] Negotiation successful
0923 17:35:20.639150 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:35:20.639822 28155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:20.622896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:46604)
0923 17:35:20.623859 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:20.623879 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:20.624023 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:20.624553 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:20.624558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:20.625267 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:20.625708 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:20.625724 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.626928 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.626933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:20.628550 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:20.628565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.629829 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.629836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:20.630010 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:20.630664 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:20.630697 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:20.631889 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:20.635467 (+ 3578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:20.635472 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:20.635476 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:20.637128 (+ 1652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:20.637579 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:20.637583 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:20.637587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:20.637780 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:20.638460 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:20.638463 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:20.638634 (+ 171us) client_negotiation.cc:769] Sending connection context
0923 17:35:20.638779 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 17:35:20.639124 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":2,"thread_start_us":682,"threads_started":1}
I20260923 17:35:20.640323 28134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:20.623222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.20.240.125:43291)
0923 17:35:20.623407 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:20.623413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:20.624656 (+ 1243us) server_negotiation.cc:407] Connection header received
0923 17:35:20.624805 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:20.624810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:20.624870 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:20.624949 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:20.625970 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.627069 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.629108 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.629831 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.629920 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:20.632972 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:20.632982 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:20.632987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:20.633023 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:20.636647 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.637370 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.637375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.637378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.637521 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.638549 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.638553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.638556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.638930 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:20.639105 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:20.639742 (+ 637us) server_negotiation.cc:299] Negotiation successful
0923 17:35:20.639965 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:35:20.641229 28156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:20.622942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:43291 (local address 127.0.0.1:51314)
0923 17:35:20.624567 (+ 1625us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:20.624581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:20.624696 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:20.624988 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:20.624992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:20.625529 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:20.625819 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:20.625832 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.627354 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.627360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:20.628960 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:20.628976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.630220 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.630227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:20.630723 (+ 496us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:20.631368 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:20.631391 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:20.632537 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:20.636771 (+ 4234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:20.636776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:20.636780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:20.637268 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:20.637636 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:20.637640 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:20.637643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:20.637791 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:20.639188 (+ 1397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:20.639193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:20.639345 (+ 152us) client_negotiation.cc:769] Sending connection context
0923 17:35:20.640481 (+ 1136us) client_negotiation.cc:240] Negotiation successful
0923 17:35:20.640784 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1500,"spinlock_wait_cycles":4992,"thread_start_us":1305,"threads_started":1}
Sep 23 17:35:20 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184918, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.126@KRBTEST.COM for oryx/127.20.240.125@KRBTEST.COM
WARNING: no policy specified for oryx/127.20.240.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.240.124@KRBTEST.COM" created.
I20260923 17:35:20.660949 28134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:20.646084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.20.240.125:43291)
0923 17:35:20.646256 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:20.646262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:20.646543 (+ 281us) server_negotiation.cc:407] Connection header received
0923 17:35:20.646886 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:20.646891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:20.646996 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:20.647054 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:20.648396 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.649148 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.650418 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.650915 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.651040 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:20.654376 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:20.654381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:20.654385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:20.654417 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:20.657437 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.658422 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.658428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.658431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.658625 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.659100 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.659105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.659108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.659497 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:20.659659 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:20.660278 (+ 619us) server_negotiation.cc:299] Negotiation successful
0923 17:35:20.660518 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:35:20.661054 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:20.645827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:43291 (local address 127.0.0.1:51328)
0923 17:35:20.646416 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:20.646465 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:20.646670 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:20.647326 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:20.647331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:20.647964 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:20.648218 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:20.648232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.649304 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.649307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:20.650280 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:20.650292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.651298 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.651303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:20.651449 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:20.652036 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:20.652055 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:20.654189 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:20.657613 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:20.657629 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:20.657638 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:20.658262 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:20.658772 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:20.658778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:20.658782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:20.658965 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:20.659672 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:20.659682 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:20.659911 (+ 229us) client_negotiation.cc:769] Sending connection context
0923 17:35:20.660174 (+ 263us) client_negotiation.cc:240] Negotiation successful
0923 17:35:20.660547 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":438,"threads_started":1}
I20260923 17:35:20.663499 27603 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.240.125:43291: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075"
member_type: NON_VOTER
last_known_addr {
host: "127.20.240.125"
port: 43291
}
attrs {
promote: true
}
}
dest_uuid: "9dc5d7a24a374aefa3e191059db69da4"
cas_config_opid_index: -1
I20260923 17:35:20.664609 27603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: NON_VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: true } }
I20260923 17:35:20.677280 28161 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 90f6ec2f0d9740be81aa4929afc03075 (127.20.240.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: NON_VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: true } } }
I20260923 17:35:20.678500 28106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:20.682227 28161 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.240.125:43291
I20260923 17:35:20.682478 28106 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 FOLLOWER]: Refusing update from remote peer 9dc5d7a24a374aefa3e191059db69da4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 17:35:20.683077 28064 master.cc:537] Master@127.20.240.125:43291 shutting down...
I20260923 17:35:20.686206 28163 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: NON_VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:20.686988 28162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: NON_VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: true } } }
I20260923 17:35:20.687381 28162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:20.695055 28064 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:20.695685 28064 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:20.695940 28064 tablet_replica.cc:330] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: stopping tablet replica
W20260923 17:35:20.696576 27587 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260923 17:35:20.698940 27587 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 -> Peer 90f6ec2f0d9740be81aa4929afc03075 (127.20.240.125:43291): Couldn't send request to peer 90f6ec2f0d9740be81aa4929afc03075. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.240.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.124.keytab.
Entry for principal oryx/127.20.240.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.124.keytab.
I20260923 17:35:20.704303 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.124:33839
--webserver_interface=127.20.240.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.124.keytab
--principal=oryx/127.20.240.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:36193,127.20.240.125:43291,127.20.240.124:33839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:35:20.714175 28064 master.cc:552] Master@127.20.240.125:43291 shutdown complete.
I20260923 17:35:20.714267 28064 master_runner.cc:305] Clearing existing system tablet
I20260923 17:35:20.714620 28064 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:20.714709 28064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:20.716929 28064 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:35:20.718312 28168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:20.718663 28064 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:20.718922 28064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:35:20.719004 28064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/wal
uuid: "90f6ec2f0d9740be81aa4929afc03075"
format_stamp: "Formatted at 2026-09-23 17:35:20 on dist-test-slave-ppjl"
server_key: "f682dd6551c0ee23e2714d76afb62963"
server_key_iv: "41fc2d0dd8338b19e7b36af0a7706140"
server_key_version: "encryptionkey@0"
I20260923 17:35:20.722627 28064 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:20.727391 28064 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 17:35:20.727571 28064 log.cc:1196] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 17:35:20.728302 28064 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: Deleting consensus metadata
I20260923 17:35:20.728569 28064 master_runner.cc:315] Copying system tablet from 127.20.240.126:36193
I20260923 17:35:20.730815 28064 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.126:36193
I20260923 17:35:20.746621 28135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:20.731674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.20.240.126:36193)
0923 17:35:20.731891 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:20.731898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:20.735731 (+ 3833us) server_negotiation.cc:407] Connection header received
0923 17:35:20.735923 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:20.735928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:20.736001 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:20.736102 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:20.736977 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.737715 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.739045 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.739548 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.739665 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:20.741490 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:20.741495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:20.741498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:20.741533 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:20.744124 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.744739 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.744744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.744746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.744867 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:20.745231 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:20.745236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:20.745239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:20.745527 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:20.745670 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:20.746147 (+ 477us) server_negotiation.cc:299] Negotiation successful
0923 17:35:20.746359 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:35:20.747545 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:20.731579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:46614)
0923 17:35:20.735592 (+ 4013us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:20.735614 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:20.735781 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:20.736143 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:20.736147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:20.736601 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:20.736819 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:20.736829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.737872 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.737877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:20.738834 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:20.738850 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:20.739696 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:20.739702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:20.739884 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:20.740506 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:20.740524 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:20.741282 (+ 758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:20.744256 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:20.744262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:20.744266 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:20.744627 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:20.744980 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:20.744985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:20.744989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:20.745127 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:20.745653 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:20.745658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:20.745837 (+ 179us) client_negotiation.cc:769] Sending connection context
0923 17:35:20.746903 (+ 1066us) client_negotiation.cc:240] Negotiation successful
0923 17:35:20.747245 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3861,"thread_start_us":3758,"threads_started":1}
I20260923 17:35:20.748347 27623 tablet_copy_service.cc:140] P 9dc5d7a24a374aefa3e191059db69da4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 90f6ec2f0d9740be81aa4929afc03075 ({username='oryx', principal='oryx/127.20.240.125@KRBTEST.COM'} at 127.0.0.1:46614)
I20260923 17:35:20.748618 27623 tablet_copy_service.cc:161] P 9dc5d7a24a374aefa3e191059db69da4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 90f6ec2f0d9740be81aa4929afc03075 at {username='oryx', principal='oryx/127.20.240.125@KRBTEST.COM'} at 127.0.0.1:46614: session id = 90f6ec2f0d9740be81aa4929afc03075-00000000000000000000000000000000
I20260923 17:35:20.751775 27623 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 17:35:20.753726 28064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 17:35:20.758895 28064 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: tablet copy: Starting download of 0 data blocks...
I20260923 17:35:20.759276 28064 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: tablet copy: Starting download of 1 WAL segments...
I20260923 17:35:20.761842 28064 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 17:35:20.765240 27623 tablet_copy_service.cc:342] P 9dc5d7a24a374aefa3e191059db69da4: Request end of tablet copy session 90f6ec2f0d9740be81aa4929afc03075-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.240.125@KRBTEST.COM'} at 127.0.0.1:46614
I20260923 17:35:20.765429 27623 tablet_copy_service.cc:433] P 9dc5d7a24a374aefa3e191059db69da4: ending tablet copy session 90f6ec2f0d9740be81aa4929afc03075-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 90f6ec2f0d9740be81aa4929afc03075
I20260923 17:35:20.775440 28064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:20.778105 28175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:20.787593 28176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:20.789019 28064 server_base.cc:1034] running on GCE node
W20260923 17:35:20.788506 28178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:20.792003 28064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:20.792666 28064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:20.793854 28064 hybrid_clock.cc:648] HybridClock initialized: now 1790184920793799 us; error 37 us; skew 500 ppm
I20260923 17:35:20.794070 28064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:20 dist-test-slave-ppjl krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184920, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:20.795862 28064 init.cc:377] Logged in from keytab as oryx/127.20.240.125@KRBTEST.COM (short username oryx)
I20260923 17:35:20.796972 28064 webserver.cc:460] Webserver started at http://127.20.240.125:35937/ using document root <none> and password file <none>
I20260923 17:35:20.797304 28064 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:20.797408 28064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:20.799693 28064 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260923 17:35:20.801285 28185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:20.801685 28064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 17:35:20.801815 28064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/wal
uuid: "90f6ec2f0d9740be81aa4929afc03075"
format_stamp: "Formatted at 2026-09-23 17:35:20 on dist-test-slave-ppjl"
server_key: "f682dd6551c0ee23e2714d76afb62963"
server_key_iv: "41fc2d0dd8338b19e7b36af0a7706140"
server_key_version: "encryptionkey@0"
I20260923 17:35:20.801957 28064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:20.816815 28064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:20.819974 28064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:20.841221 28064 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.125:43291
I20260923 17:35:20.841243 28236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.125:43291 every 8 connection(s)
I20260923 17:35:20.842127 28064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-1/data/info.pb
I20260923 17:35:20.847687 28237 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:35:20.852082 28237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: Bootstrap starting.
I20260923 17:35:20.867965 28237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:35:20.868402 28237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: Bootstrap complete.
I20260923 17:35:20.869232 28237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: NON_VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: true } }
I20260923 17:35:20.869438 28237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90f6ec2f0d9740be81aa4929afc03075, State: Initialized, Role: LEARNER
I20260923 17:35:20.869669 28237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: NON_VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: true } }
I20260923 17:35:20.870429 28240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: NON_VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: true } } }
I20260923 17:35:20.870724 28237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:35:20.870893 28240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: This master's current role is: LEARNER
I20260923 17:35:20.880640 28251 catalog_manager.cc:1259] Loaded cluster ID: f902ec2a20da4b6f917072be6129bc21
I20260923 17:35:20.880764 28251 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: loading cluster ID for follower catalog manager: success
I20260923 17:35:20.883471 28251 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: acquiring CA information for follower catalog manager: success
I20260923 17:35:20.885390 28251 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:35:21.055456 28059 heartbeater.cc:503] Master 127.20.240.126:36193 was elected leader, sending a full tablet report...
W20260923 17:35:21.133054 28166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:21.133410 28166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:21.133560 28166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:21.141404 28166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:35:21.141526 28166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:21.141577 28166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:21.141654 28166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:35:21.141708 28166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:35:21.152184 28166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:36193,127.20.240.125:43291,127.20.240.124:33839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.124.keytab
--principal=oryx/127.20.240.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.124:33839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.240.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:21.153534 28166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:21.154853 28166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:21.167739 28257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:21.167752 28259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:21.167652 28256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:21.168466 28166 server_base.cc:1034] running on GCE node
I20260923 17:35:21.169266 28166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:21.170739 28166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:21.171938 28166 hybrid_clock.cc:648] HybridClock initialized: now 1790184921171886 us; error 42 us; skew 500 ppm
I20260923 17:35:21.172318 28166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:21 dist-test-slave-ppjl krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184921, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:21.175436 28166 init.cc:377] Logged in from keytab as oryx/127.20.240.124@KRBTEST.COM (short username oryx)
I20260923 17:35:21.177017 28166 webserver.cc:460] Webserver started at http://127.20.240.124:34747/ using document root <none> and password file <none>
I20260923 17:35:21.177757 28166 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:21.177875 28166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:21.178184 28166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:21.181342 28166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/data/instance:
uuid: "92f052fe126b402c805a99e40f9296c0"
format_stamp: "Formatted at 2026-09-23 17:35:21 on dist-test-slave-ppjl"
server_key: "8b4b047d4903945db50ba44e0395fcba"
server_key_iv: "a142db03bae0fcc2f38854e4733dfe27"
server_key_version: "encryptionkey@0"
I20260923 17:35:21.182116 28166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/wal/instance:
uuid: "92f052fe126b402c805a99e40f9296c0"
format_stamp: "Formatted at 2026-09-23 17:35:21 on dist-test-slave-ppjl"
server_key: "8b4b047d4903945db50ba44e0395fcba"
server_key_iv: "a142db03bae0fcc2f38854e4733dfe27"
server_key_version: "encryptionkey@0"
I20260923 17:35:21.187811 28166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:35:21.191406 28266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:21.192853 28166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:35:21.193018 28166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/wal
uuid: "92f052fe126b402c805a99e40f9296c0"
format_stamp: "Formatted at 2026-09-23 17:35:21 on dist-test-slave-ppjl"
server_key: "8b4b047d4903945db50ba44e0395fcba"
server_key_iv: "a142db03bae0fcc2f38854e4733dfe27"
server_key_version: "encryptionkey@0"
I20260923 17:35:21.193217 28166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:21.221848 28166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:21.226670 28166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:21.226953 28166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:21.249563 28166 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.124:33839
I20260923 17:35:21.249765 28320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.124:33839 every 8 connection(s)
I20260923 17:35:21.251371 28166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/data/info.pb
I20260923 17:35:21.253816 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.245101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:43291 (local address 127.0.0.1:51334)
0923 17:35:21.245505 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:21.245525 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:21.245667 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:21.246500 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:21.246505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:21.246536 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:21.246815 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.246826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.249010 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.249014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.250887 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.250903 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.252712 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.252719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.253000 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.253073 (+ 73us) client_negotiation.cc:769] Sending connection context
0923 17:35:21.253181 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 17:35:21.253346 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":208,"threads_started":1}
I20260923 17:35:21.254212 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.245299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51334 (local address 127.20.240.125:43291)
0923 17:35:21.246147 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:21.246152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:21.246176 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:35:21.246240 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:21.246243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:21.246313 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:21.246421 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:21.246978 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.248834 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.251115 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.252472 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.252706 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.252919 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:21.253542 (+ 623us) server_negotiation.cc:299] Negotiation successful
0923 17:35:21.253709 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":592,"threads_started":1}
I20260923 17:35:21.255743 28212 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260923 17:35:21.256440 28240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dc5d7a24a374aefa3e191059db69da4. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: NON_VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: true } } }
I20260923 17:35:21.256819 28240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: This master's current role is: LEARNER
I20260923 17:35:21.258042 28321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:21.258308 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 28166
I20260923 17:35:21.268945 28321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:35:21 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.240.124@KRBTEST.COM
Sep 23 17:35:21 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184921, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.124@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
I20260923 17:35:21.284734 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.260987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.20.240.124:33839)
0923 17:35:21.262603 (+ 1616us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:21.262616 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:21.262680 (+ 64us) server_negotiation.cc:407] Connection header received
0923 17:35:21.264413 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:21.264439 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:21.264853 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:21.265266 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:21.266808 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.268164 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.269508 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.270234 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.270381 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.275875 (+ 5494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:21.275888 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:21.275906 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:21.275952 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:21.279549 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.280537 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.280545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.280552 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.280756 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.281237 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.281244 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.281247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.281900 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:21.282212 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:21.282658 (+ 446us) server_negotiation.cc:299] Negotiation successful
0923 17:35:21.283101 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1380,"thread_start_us":1089,"threads_started":1}
I20260923 17:35:21.288666 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.272276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46630 (local address 127.20.240.126:36193)
0923 17:35:21.272840 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:21.272847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:21.272870 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:35:21.272988 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:21.272994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:21.273095 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:21.273222 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:21.274586 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.275690 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.277506 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.278167 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.278302 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.282122 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:21.282130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:21.282134 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:21.282178 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:21.285463 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.286387 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.286393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.286396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.286576 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.286954 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.286958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.286961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.287399 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:21.287570 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:21.288169 (+ 599us) server_negotiation.cc:299] Negotiation successful
0923 17:35:21.288365 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":853,"threads_started":1}
I20260923 17:35:21.288733 28325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.271874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:46630)
0923 17:35:21.272680 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:21.272735 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:21.272881 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:21.273296 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:21.273301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:21.273952 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:21.274394 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.274409 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.275861 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.275867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.277355 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.277372 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.278364 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.278371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.278547 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.279195 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:21.279221 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:21.281898 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:21.285645 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.285655 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.285662 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.286254 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.286698 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.286702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.286704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.286851 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.287570 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:21.287577 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:21.287818 (+ 241us) client_negotiation.cc:769] Sending connection context
0923 17:35:21.288033 (+ 215us) client_negotiation.cc:240] Negotiation successful
0923 17:35:21.288365 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"spinlock_wait_cycles":17792,"thread_start_us":563,"threads_started":1}
I20260923 17:35:21.289997 28321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:35:21 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184921, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.124@KRBTEST.COM for oryx/127.20.240.125@KRBTEST.COM
I20260923 17:35:21.303139 28325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.290589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:43291 (local address 127.0.0.1:51344)
0923 17:35:21.290742 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:21.290757 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:21.290834 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:21.291341 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:21.291344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:21.291709 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:21.291862 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.291870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.292901 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.292905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.294012 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.294029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.294794 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.294799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.294957 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.295540 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:21.295557 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:21.297642 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:21.300665 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.300669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.300672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.301121 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.301535 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.301539 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.301541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.301692 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.302323 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:21.302327 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:21.302468 (+ 141us) client_negotiation.cc:769] Sending connection context
0923 17:35:21.302578 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 17:35:21.302872 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 17:35:21.303257 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.290913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.20.240.125:43291)
0923 17:35:21.291060 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:21.291065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:21.291080 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:35:21.291137 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:21.291140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:21.291226 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:21.291299 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:21.291967 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.292804 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.294212 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.294638 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.294736 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.297858 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:21.297863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:21.297866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:21.297915 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:21.300532 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.301259 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.301264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.301266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.301438 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.301802 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.301806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.301808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.302214 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:21.302334 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:21.302698 (+ 364us) server_negotiation.cc:299] Negotiation successful
0923 17:35:21.302900 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:35:21.304713 28321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:35:21 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184921, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.124@KRBTEST.COM for oryx/127.20.240.124@KRBTEST.COM
I20260923 17:35:21.318302 28325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.305556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:33839 (local address 127.0.0.1:35944)
0923 17:35:21.305728 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:21.305747 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:21.305874 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:21.306237 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:21.306241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:21.306667 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:21.306840 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.306849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.308004 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.308009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.308904 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.308917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.309631 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.309637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.309883 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.310557 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:21.310581 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:21.312756 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:21.315675 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.315681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.315685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.316161 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.316523 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.316529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.316533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.316744 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.317280 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:21.317285 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:21.317424 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 17:35:21.317560 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 17:35:21.317828 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 17:35:21.318337 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.305525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.20.240.124:33839)
0923 17:35:21.305728 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:21.305734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:21.305814 (+ 80us) server_negotiation.cc:407] Connection header received
0923 17:35:21.306058 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:21.306061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:21.306121 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:21.306201 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:21.306984 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.307875 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.309047 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.309462 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.309587 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.312942 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:21.312950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:21.312954 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:21.312983 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:21.315541 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.316278 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.316284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.316287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.316410 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.316851 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.316856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.316860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.317175 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:21.317303 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:21.317651 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 17:35:21.318009 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":64}
I20260923 17:35:21.323997 28321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: Bootstrap starting.
I20260923 17:35:21.327527 28321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:21.328703 28321 log.cc:824] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:21.331964 28321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: No bootstrap required, opened a new log
I20260923 17:35:21.339684 28321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } }
I20260923 17:35:21.340039 28321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:21.340128 28321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f052fe126b402c805a99e40f9296c0, State: Initialized, Role: FOLLOWER
I20260923 17:35:21.340658 28321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } }
I20260923 17:35:21.341516 28328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } } }
I20260923 17:35:21.341845 28328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:35:21.342558 28321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:35:21.351655 28339 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:35:21.351823 28339 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:35:21.357620 28166 master_runner.cc:428] Detected that this master 92f052fe126b402c805a99e40f9296c0 is joining an existing cluster
I20260923 17:35:21.357738 28166 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.240.124:33839
I20260923 17:35:21.364240 28328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:35:21.364607 28328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } }
I20260923 17:35:21.366466 28328 leader_election.cc:288] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dc5d7a24a374aefa3e191059db69da4 (127.20.240.126:36193), 90f6ec2f0d9740be81aa4929afc03075 (127.20.240.125:43291)
I20260923 17:35:21.367304 27613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92f052fe126b402c805a99e40f9296c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dc5d7a24a374aefa3e191059db69da4" is_pre_election: true
I20260923 17:35:21.367381 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92f052fe126b402c805a99e40f9296c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90f6ec2f0d9740be81aa4929afc03075" is_pre_election: true
I20260923 17:35:21.368821 28269 leader_election.cc:302] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92f052fe126b402c805a99e40f9296c0; no voters: 90f6ec2f0d9740be81aa4929afc03075, 9dc5d7a24a374aefa3e191059db69da4
I20260923 17:35:21.369649 28328 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 17:35:21.392781 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 27664
I20260923 17:35:21.393918 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.363020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51360 (local address 127.20.240.125:43291)
0923 17:35:21.363221 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:21.363227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:21.376529 (+ 13302us) server_negotiation.cc:407] Connection header received
0923 17:35:21.377009 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:21.377016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:21.377099 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:21.377168 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:21.379249 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.380311 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.382682 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.383350 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.383460 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.386890 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:21.386898 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:21.386903 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:21.386946 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:21.390490 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.391393 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.391399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.391404 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.391581 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.392075 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.392080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.392084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.392448 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:21.392584 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:21.393290 (+ 706us) server_negotiation.cc:299] Negotiation successful
0923 17:35:21.393557 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:35:21.395231 28347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.362568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:43291 (local address 127.0.0.1:51360)
0923 17:35:21.376410 (+ 13842us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:21.376427 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:21.376838 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:21.377377 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:21.377381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:21.378495 (+ 1114us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:21.379031 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.379044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.380507 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.380513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.382504 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.382523 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.383831 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.383838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.384042 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.384886 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:21.384908 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:21.386653 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:21.390675 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.390681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.390684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.391228 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.391746 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.391751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.391754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.391941 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.392659 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:21.392663 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:21.392898 (+ 235us) client_negotiation.cc:769] Sending connection context
0923 17:35:21.394119 (+ 1221us) client_negotiation.cc:240] Negotiation successful
0923 17:35:21.394621 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13688,"mutex_wait_us":155,"spinlock_wait_cycles":66688,"thread_start_us":12930,"threads_started":1}
I20260923 17:35:21.399572 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.362319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.20.240.124:33839)
0923 17:35:21.362560 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:21.362567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:21.376775 (+ 14208us) server_negotiation.cc:407] Connection header received
0923 17:35:21.376934 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:21.376939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:21.377006 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:21.377097 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:21.379336 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.380349 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.382608 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.383320 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.383414 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.386776 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:21.386783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:21.386787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:21.386825 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:21.394651 (+ 7826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.397257 (+ 2606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.397263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.397266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.397428 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.397894 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.397899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.397902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.398256 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:21.398415 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:21.398956 (+ 541us) server_negotiation.cc:299] Negotiation successful
0923 17:35:21.399208 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:35:21.400478 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.362933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:33839 (local address 127.0.0.1:35952)
0923 17:35:21.376670 (+ 13737us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:21.376685 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:21.376822 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:21.377308 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:21.377313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:21.378506 (+ 1193us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:21.379205 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.379218 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.380565 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.380569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.382465 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.382474 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.383760 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.383767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.384012 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.384996 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:21.385016 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:21.386584 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:21.394809 (+ 8225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.394814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.394818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.397126 (+ 2308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.397642 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.397647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.397650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.397793 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.398496 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:21.398500 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:21.398702 (+ 202us) client_negotiation.cc:769] Sending connection context
0923 17:35:21.399803 (+ 1101us) client_negotiation.cc:240] Negotiation successful
0923 17:35:21.400085 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13604,"mutex_wait_us":162,"thread_start_us":13619,"threads_started":1}
I20260923 17:35:21.400527 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.362946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46634 (local address 127.20.240.126:36193)
0923 17:35:21.363135 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:21.363139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:21.376117 (+ 12978us) server_negotiation.cc:407] Connection header received
0923 17:35:21.376345 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:21.376351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:21.376443 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:21.376572 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:21.379045 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.380231 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.382626 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.383320 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.383414 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.386799 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:21.386806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:21.386809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:21.386850 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:21.391901 (+ 5051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.392650 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.392655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.392659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.392890 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.393350 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.393356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.393359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.394895 (+ 1536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:21.395041 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:21.399952 (+ 4911us) server_negotiation.cc:299] Negotiation successful
0923 17:35:21.400237 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:35:21.401546 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.362466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:46634)
0923 17:35:21.375964 (+ 13498us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:21.375990 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:21.376192 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:21.376764 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:21.376769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:21.377704 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:21.378822 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.378841 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.380451 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.380455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.382458 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.382474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.383707 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.383714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.383922 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.384749 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:21.384772 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:21.386562 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:21.392052 (+ 5490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.392056 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.392059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.392512 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.393041 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.393046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.393049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.393231 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.399301 (+ 6070us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:21.399307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:21.399582 (+ 275us) client_negotiation.cc:769] Sending connection context
0923 17:35:21.400761 (+ 1179us) client_negotiation.cc:240] Negotiation successful
0923 17:35:21.401071 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13219,"mutex_wait_us":229,"thread_start_us":13878,"threads_started":1}
Sep 23 17:35:21 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184918, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.126@KRBTEST.COM for oryx/127.20.240.124@KRBTEST.COM
Sep 23 17:35:21 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.240.65@KRBTEST.COM
I20260923 17:35:21.421788 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.406777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:33839 (local address 127.0.0.1:35960)
0923 17:35:21.407022 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:21.407040 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:21.407145 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:21.407509 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:21.407514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:21.407972 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:21.408163 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.408175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.409686 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.409691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.410937 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.410953 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.411848 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.411854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.412054 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.412584 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:21.412602 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:21.414727 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:21.418830 (+ 4103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.418836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.418840 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.419408 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.419906 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.419911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.419915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.420113 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.420803 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:21.420808 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:21.420980 (+ 172us) client_negotiation.cc:769] Sending connection context
0923 17:35:21.421117 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 17:35:21.421412 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260923 17:35:21.423237 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.406855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35960 (local address 127.20.240.124:33839)
0923 17:35:21.407012 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:21.407021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:21.407084 (+ 63us) server_negotiation.cc:407] Connection header received
0923 17:35:21.407309 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:21.407314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:21.407382 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:21.407469 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:21.408338 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.409549 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.411091 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.411693 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.411804 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.415011 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:21.415016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:21.415019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:21.415053 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:21.418644 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.419601 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.419606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.419609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.419762 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.420270 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.420275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.420278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.420646 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:21.422504 (+ 1858us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:21.422648 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 17:35:21.422889 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:35:21.424512 28348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.393926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34316 (local address 127.20.240.65:37703)
0923 17:35:21.400373 (+ 6447us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:21.400382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:21.400412 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:35:21.400485 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:21.400493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:21.403548 (+ 3055us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:21.405564 (+ 2016us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:21.405833 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.407444 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.409053 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.409974 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.410102 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.416187 (+ 6085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:21.416209 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:21.416231 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:21.416278 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:21.420322 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.421289 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.421299 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.421302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.421452 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.421920 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.421925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.421929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.422358 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:21.422648 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:21.423838 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0923 17:35:21.424116 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6264,"thread_start_us":6076,"threads_started":1}
I20260923 17:35:21.425165 27603 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.240.124:33839: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "92f052fe126b402c805a99e40f9296c0"
member_type: NON_VOTER
last_known_addr {
host: "127.20.240.124"
port: 33839
}
attrs {
promote: true
}
}
dest_uuid: "9dc5d7a24a374aefa3e191059db69da4"
cas_config_opid_index: 5
I20260923 17:35:21.426276 27603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: NON_VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: true } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } }
I20260923 17:35:21.426607 27787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:21.428606 28308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 92f052fe126b402c805a99e40f9296c0 (127.20.240.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: NON_VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: true } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } } }
I20260923 17:35:21.432123 28212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 LEARNER]: Refusing update from remote peer 9dc5d7a24a374aefa3e191059db69da4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 17:35:21.434198 28293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:21.438308 28293 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 1 FOLLOWER]: Refusing update from remote peer 9dc5d7a24a374aefa3e191059db69da4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 17:35:21.446516 28352 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: NON_VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 17:35:21.451884 28308 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.240.124:33839
I20260923 17:35:21.452751 28166 master.cc:537] Master@127.20.240.124:33839 shutting down...
I20260923 17:35:21.454998 28352 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:21.461790 28351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: NON_VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: true } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } } }
I20260923 17:35:21.462355 28351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:21.473114 28212 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 92f052fe126b402c805a99e40f9296c0 (127.20.240.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: NON_VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: true } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } } }
I20260923 17:35:21.477396 28240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: NON_VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: true } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } } }
I20260923 17:35:21.477705 28240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: This master's current role is: LEARNER
I20260923 17:35:21.478761 28352 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4: attempting to promote NON_VOTER 90f6ec2f0d9740be81aa4929afc03075 to VOTER
I20260923 17:35:21.479806 28352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } }
I20260923 17:35:21.482391 28212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 LEARNER]: Refusing update from remote peer 9dc5d7a24a374aefa3e191059db69da4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 17:35:21.482936 28352 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 17:35:21.485186 28351 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 90f6ec2f0d9740be81aa4929afc03075 (127.20.240.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } } }
I20260923 17:35:21.488157 28351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } } }
I20260923 17:35:21.488440 28351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:21.488937 28308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } } }
I20260923 17:35:21.489198 28308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:21.489661 28212 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 90f6ec2f0d9740be81aa4929afc03075 (127.20.240.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } } }
W20260923 17:35:21.492182 28272 tablet.cc:2369] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:35:21.493106 28240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } } }
I20260923 17:35:21.493362 28240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 17:35:21.496114 27590 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 -> Peer 92f052fe126b402c805a99e40f9296c0 (127.20.240.124:33839): Couldn't send request to peer 92f052fe126b402c805a99e40f9296c0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 17:35:21.496413 28166 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:21.496783 28166 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:21.496997 28166 tablet_replica.cc:330] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: stopping tablet replica
I20260923 17:35:21.502919 28166 master.cc:552] Master@127.20.240.124:33839 shutdown complete.
I20260923 17:35:21.503015 28166 master_runner.cc:305] Clearing existing system tablet
I20260923 17:35:21.503407 28166 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:21.503520 28166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:21.505712 28166 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 17:35:21.507031 28357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:21.507395 28166 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:21.507644 28166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 17:35:21.507738 28166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/wal
uuid: "92f052fe126b402c805a99e40f9296c0"
format_stamp: "Formatted at 2026-09-23 17:35:21 on dist-test-slave-ppjl"
server_key: "8b4b047d4903945db50ba44e0395fcba"
server_key_iv: "a142db03bae0fcc2f38854e4733dfe27"
server_key_version: "encryptionkey@0"
I20260923 17:35:21.511051 28166 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:21.515731 28166 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 17:35:21.515879 28166 log.cc:1196] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 17:35:21.516253 28166 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: Deleting consensus metadata
I20260923 17:35:21.516479 28166 master_runner.cc:315] Copying system tablet from 127.20.240.126:36193
I20260923 17:35:21.518812 28166 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.126:36193
I20260923 17:35:21.535683 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.519623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46638 (local address 127.20.240.126:36193)
0923 17:35:21.519898 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:21.519905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:21.523730 (+ 3825us) server_negotiation.cc:407] Connection header received
0923 17:35:21.523948 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:21.523952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:21.524029 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:21.524114 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:21.525004 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.525754 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.527023 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.527554 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.527672 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.529641 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:21.529647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:21.529650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:21.529687 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:21.532582 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.533283 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.533289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.533293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.533436 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:21.533918 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:21.533923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:21.533926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:21.534306 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:21.534436 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:21.535034 (+ 598us) server_negotiation.cc:299] Negotiation successful
0923 17:35:21.535312 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260923 17:35:21.536731 28363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.519528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:46638)
0923 17:35:21.523602 (+ 4074us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:21.523624 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:21.523798 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:21.524166 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:21.524171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:21.524628 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:21.524844 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.524855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.525912 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.525916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.526879 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.526892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.527744 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.527751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.527929 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.528584 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:21.528603 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:21.529425 (+ 822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:21.532736 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.532742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.532746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.533160 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.533557 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:21.533562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:21.533566 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:21.533778 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:21.534547 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:21.534552 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:21.534735 (+ 183us) client_negotiation.cc:769] Sending connection context
0923 17:35:21.535941 (+ 1206us) client_negotiation.cc:240] Negotiation successful
0923 17:35:21.536293 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3914,"thread_start_us":3800,"threads_started":1}
I20260923 17:35:21.537464 27623 tablet_copy_service.cc:140] P 9dc5d7a24a374aefa3e191059db69da4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 92f052fe126b402c805a99e40f9296c0 ({username='oryx', principal='oryx/127.20.240.124@KRBTEST.COM'} at 127.0.0.1:46638)
I20260923 17:35:21.537645 27623 tablet_copy_service.cc:161] P 9dc5d7a24a374aefa3e191059db69da4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 92f052fe126b402c805a99e40f9296c0 at {username='oryx', principal='oryx/127.20.240.124@KRBTEST.COM'} at 127.0.0.1:46638: session id = 92f052fe126b402c805a99e40f9296c0-00000000000000000000000000000000
I20260923 17:35:21.540093 27623 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 17:35:21.541779 28166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 17:35:21.547137 28166 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: tablet copy: Starting download of 0 data blocks...
I20260923 17:35:21.547464 28166 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: tablet copy: Starting download of 1 WAL segments...
I20260923 17:35:21.549705 28166 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 17:35:21.553175 27623 tablet_copy_service.cc:342] P 9dc5d7a24a374aefa3e191059db69da4: Request end of tablet copy session 92f052fe126b402c805a99e40f9296c0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.240.124@KRBTEST.COM'} at 127.0.0.1:46638
I20260923 17:35:21.553309 27623 tablet_copy_service.cc:433] P 9dc5d7a24a374aefa3e191059db69da4: ending tablet copy session 92f052fe126b402c805a99e40f9296c0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 92f052fe126b402c805a99e40f9296c0
I20260923 17:35:21.563941 28166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:21.567516 28364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:21.575479 28365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:21.578692 28166 server_base.cc:1034] running on GCE node
W20260923 17:35:21.578714 28368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:21.579362 28166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:21.579977 28166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:21.581151 28166 hybrid_clock.cc:648] HybridClock initialized: now 1790184921581118 us; error 30 us; skew 500 ppm
I20260923 17:35:21.581395 28166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:21 dist-test-slave-ppjl krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184921, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:21.583388 28166 init.cc:377] Logged in from keytab as oryx/127.20.240.124@KRBTEST.COM (short username oryx)
I20260923 17:35:21.588141 28166 webserver.cc:460] Webserver started at http://127.20.240.124:41617/ using document root <none> and password file <none>
I20260923 17:35:21.588510 28166 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:21.588621 28166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:21.591339 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 27664
I20260923 17:35:21.591928 28166 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 17:35:21.593838 28374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:21.594233 28166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260923 17:35:21.594341 28166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/wal
uuid: "92f052fe126b402c805a99e40f9296c0"
format_stamp: "Formatted at 2026-09-23 17:35:21 on dist-test-slave-ppjl"
server_key: "8b4b047d4903945db50ba44e0395fcba"
server_key_iv: "a142db03bae0fcc2f38854e4733dfe27"
server_key_version: "encryptionkey@0"
I20260923 17:35:21.594527 28166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:21.608692 28166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:21.612700 28166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:21.613044 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:37703
--local_ip_for_outbound_sockets=127.20.240.65
--tserver_master_addrs=127.20.240.126:36193
--webserver_port=34621
--webserver_interface=127.20.240.65
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.65.keytab
--principal=oryx/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.240.126:36193,127.20.240.125:43291
--tserver_master_addrs=127.20.240.126:36193,127.20.240.125:43291,127.20.240.124:33839 with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:35:21.631434 28166 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.124:33839
I20260923 17:35:21.631467 28426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.124:33839 every 8 connection(s)
I20260923 17:35:21.632547 28166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/master-2/data/info.pb
I20260923 17:35:21.637890 28427 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:35:21.641685 28427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: Bootstrap starting.
I20260923 17:35:21.660544 28427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:35:21.661077 28427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: Bootstrap complete.
I20260923 17:35:21.662103 28427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } }
I20260923 17:35:21.662400 28427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 92f052fe126b402c805a99e40f9296c0, State: Initialized, Role: LEARNER
I20260923 17:35:21.662632 28427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } }
I20260923 17:35:21.663697 28427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:35:21.663455 28430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } } }
I20260923 17:35:21.663980 28430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [sys.catalog]: This master's current role is: LEARNER
I20260923 17:35:21.688169 28441 catalog_manager.cc:1259] Loaded cluster ID: f902ec2a20da4b6f917072be6129bc21
I20260923 17:35:21.688303 28441 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: loading cluster ID for follower catalog manager: success
I20260923 17:35:21.691962 28441 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: acquiring CA information for follower catalog manager: success
I20260923 17:35:21.694901 28441 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:35:21.925465 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.919548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:33839 (local address 127.0.0.1:41304)
0923 17:35:21.919751 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:21.919768 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:21.919911 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:21.920653 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:21.920689 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:21.920722 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:21.920892 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.920903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.922292 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.922296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.923490 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:21.923505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.924645 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.924652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:21.924866 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.924901 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:35:21.924991 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 17:35:21.925144 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 17:35:21.925485 28442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:21.919765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.20.240.124:33839)
0923 17:35:21.920336 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:21.920340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:21.920356 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:35:21.920417 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:21.920420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:21.920489 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:21.920596 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:21.921048 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.922127 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.923651 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:21.924479 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:21.924656 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:21.924870 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:21.925025 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 17:35:21.925154 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":312,"threads_started":1}
I20260923 17:35:21.926460 28402 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260923 17:35:21.927028 28430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dc5d7a24a374aefa3e191059db69da4. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } } }
I20260923 17:35:21.927369 28430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [sys.catalog]: This master's current role is: LEARNER
W20260923 17:35:21.978952 28393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:21.979280 28393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:21.979454 28393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:21.987705 28393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:21.987843 28393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:21.988015 28393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:35:21.999426 28393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.65.keytab
--principal=oryx/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:37703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=34621
--tserver_master_addrs=127.20.240.126:36193,127.20.240.125:43291,127.20.240.124:33839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:22.000922 28393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:22.002411 28393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:22.017103 28448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:22.017230 28447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:22.017099 28450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:22.018184 28393 server_base.cc:1034] running on GCE node
I20260923 17:35:22.018821 28393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:22.019939 28393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:22.021180 28393 hybrid_clock.cc:648] HybridClock initialized: now 1790184922021106 us; error 44 us; skew 500 ppm
I20260923 17:35:22.021516 28393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:22 dist-test-slave-ppjl krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184922, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:22.025096 28393 init.cc:377] Logged in from keytab as oryx/127.20.240.65@KRBTEST.COM (short username oryx)
I20260923 17:35:22.026953 28393 webserver.cc:460] Webserver started at http://127.20.240.65:34621/ using document root <none> and password file <none>
I20260923 17:35:22.027779 28393 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:22.027952 28393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:22.033981 28393 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:35:22.037290 28457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:22.038832 28393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:35:22.039000 28393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "4dbaafbf88034b6da6cfd705d4849bba"
format_stamp: "Formatted at 2026-09-23 17:35:18 on dist-test-slave-ppjl"
server_key: "3ab5bb85d6a25fbd8ddcd394a4d72e98"
server_key_iv: "e6cbb8563101f9ca30f8528440d69711"
server_key_version: "encryptionkey@0"
I20260923 17:35:22.040107 28393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:22.072180 28393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:22.077535 28393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:22.077868 28393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:22.079080 28393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:22.080852 28393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:22.080996 28393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:22.081142 28393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:22.081244 28393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:22.119006 28393 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:37703
I20260923 17:35:22.119062 28569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:37703 every 8 connection(s)
I20260923 17:35:22.120473 28393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:35:22.131114 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 28393
I20260923 17:35:22.131527 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 27798
Sep 23 17:35:22 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184922, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
Sep 23 17:35:22 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184922, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.124@KRBTEST.COM
I20260923 17:35:22.161798 28578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:22.124635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.65:54107)
0923 17:35:22.128776 (+ 4141us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:22.128850 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:22.132592 (+ 3742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:22.133943 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:22.133960 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:22.134980 (+ 1020us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:22.136053 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:22.136083 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.137822 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.137828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:22.140341 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:22.140360 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.141972 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.141980 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:22.142350 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:22.146274 (+ 3924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:22.146320 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:22.150118 (+ 3798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:22.156422 (+ 6304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:22.156431 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:22.156451 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:22.157118 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:22.157589 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:22.157594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:22.157598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:22.157881 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:22.158643 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:22.158652 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:22.159283 (+ 631us) client_negotiation.cc:769] Sending connection context
0923 17:35:22.159626 (+ 343us) client_negotiation.cc:240] Negotiation successful
0923 17:35:22.160244 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3738,"thread_start_us":24467,"threads_started":1}
Sep 23 17:35:22 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184922, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.125@KRBTEST.COM
I20260923 17:35:22.161980 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:22.124059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:54107 (local address 127.20.240.126:36193)
0923 17:35:22.126081 (+ 2022us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:22.126088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:22.131681 (+ 5593us) server_negotiation.cc:407] Connection header received
0923 17:35:22.133108 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:22.133116 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:22.133243 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:22.133351 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:22.136329 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.137569 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.140568 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.141406 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.141537 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:22.152558 (+ 11021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:22.152569 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:22.152575 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:22.152620 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:22.156244 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.157278 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.157284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.157287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.157455 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.158040 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.158045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.158048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.158400 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:22.158558 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:22.161369 (+ 2811us) server_negotiation.cc:299] Negotiation successful
0923 17:35:22.161625 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1878,"thread_start_us":1760,"threads_started":1}
I20260923 17:35:22.164008 28571 heartbeater.cc:347] Connected to a master server at 127.20.240.126:36193
I20260923 17:35:22.164378 28571 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:22.165352 28571 heartbeater.cc:511] Master 127.20.240.126:36193 requested a full tablet report, sending...
Sep 23 17:35:22 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.240.66@KRBTEST.COM
I20260923 17:35:22.169589 27603 ts_manager.cc:194] Re-registered known tserver with Master: 4dbaafbf88034b6da6cfd705d4849bba (127.20.240.65:37703)
I20260923 17:35:22.170746 27603 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:54107
I20260923 17:35:22.173368 28442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:22.123888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:42079 (local address 127.20.240.124:33839)
0923 17:35:22.124513 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:22.124520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:22.151635 (+ 27115us) server_negotiation.cc:407] Connection header received
0923 17:35:22.151713 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:22.151718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:22.151806 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:22.151917 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:22.153361 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.154568 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.156608 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.157304 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.157395 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:22.163182 (+ 5787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:22.163190 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:22.163195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:22.163243 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:22.168357 (+ 5114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.170342 (+ 1985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.170349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.170352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.170501 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.171205 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.171210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.171214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.171564 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:22.171718 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:22.172733 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0923 17:35:22.173010 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459}
I20260923 17:35:22.173532 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:22.124122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:60697 (local address 127.20.240.125:43291)
0923 17:35:22.128232 (+ 4110us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:22.128239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:22.148987 (+ 20748us) server_negotiation.cc:407] Connection header received
0923 17:35:22.150314 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:22.150321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:22.150432 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:22.150515 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:22.152071 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.153184 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.155046 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.158158 (+ 3112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.158262 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:22.164821 (+ 6559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:22.164829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:22.164834 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:22.164877 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:22.168412 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.171364 (+ 2952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.171370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.171373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.171532 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.171946 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.171952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.171956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.172317 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:22.172432 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:22.172918 (+ 486us) server_negotiation.cc:299] Negotiation successful
0923 17:35:22.173161 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3980,"thread_start_us":3835,"threads_started":1}
I20260923 17:35:22.174674 28577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:22.124875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:43291 (local address 127.20.240.65:60697)
0923 17:35:22.148843 (+ 23968us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:22.148866 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:22.150135 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:22.150808 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:22.150813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:22.151564 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:22.151893 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:22.151906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.153374 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.153379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:22.154861 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:22.154876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.158568 (+ 3692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.158574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:22.158770 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:22.159682 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:22.159704 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:22.164570 (+ 4866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:22.168592 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:22.168597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:22.168602 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:22.170648 (+ 2046us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:22.171670 (+ 1022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:22.171675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:22.171680 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:22.171831 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:22.172521 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:22.172525 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:22.172672 (+ 147us) client_negotiation.cc:769] Sending connection context
0923 17:35:22.173721 (+ 1049us) client_negotiation.cc:240] Negotiation successful
0923 17:35:22.174035 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23821,"thread_start_us":24998,"threads_started":1}
I20260923 17:35:22.175333 28580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:22.132606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.20.240.66:35803)
0923 17:35:22.147678 (+ 15072us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:22.147688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:22.151783 (+ 4095us) server_negotiation.cc:407] Connection header received
0923 17:35:22.152227 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:22.152238 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:22.152532 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:22.152701 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:22.155576 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.157093 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.159470 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.160361 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.160461 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:22.167783 (+ 7322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:22.167802 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:22.167819 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:22.167864 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:22.171614 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.172632 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.172642 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.172645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.172800 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.173359 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.173364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.173367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.173783 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:22.174038 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:22.174638 (+ 600us) server_negotiation.cc:299] Negotiation successful
0923 17:35:22.174900 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14860,"thread_start_us":15082,"threads_started":1}
I20260923 17:35:22.175540 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:22.124635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:33839 (local address 127.20.240.65:42079)
0923 17:35:22.149560 (+ 24925us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:22.149576 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:22.149776 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:22.151951 (+ 2175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:22.151956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:22.152485 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:22.152792 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:22.152806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.154735 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.154740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:22.156434 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:22.156446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.157695 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.157701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:22.157882 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:22.158666 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:22.158687 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:22.162890 (+ 4203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:22.168828 (+ 5938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:22.168833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:22.168836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:22.170191 (+ 1355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:22.170803 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:22.170807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:22.170811 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:22.171047 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:22.172294 (+ 1247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:22.172299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:22.172456 (+ 157us) client_negotiation.cc:769] Sending connection context
0923 17:35:22.174230 (+ 1774us) client_negotiation.cc:240] Negotiation successful
0923 17:35:22.174989 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24775,"mutex_wait_us":350,"thread_start_us":3207,"threads_started":1}
I20260923 17:35:22.176596 28570 heartbeater.cc:347] Connected to a master server at 127.20.240.124:33839
I20260923 17:35:22.176693 28572 heartbeater.cc:347] Connected to a master server at 127.20.240.125:43291
I20260923 17:35:22.176752 28570 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:22.176828 28572 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:22.177531 27921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:22.177781 28570 heartbeater.cc:511] Master 127.20.240.124:33839 requested a full tablet report, sending...
I20260923 17:35:22.178296 28572 heartbeater.cc:511] Master 127.20.240.125:43291 requested a full tablet report, sending...
I20260923 17:35:22.180250 28391 ts_manager.cc:194] Registered new tserver with Master: 4dbaafbf88034b6da6cfd705d4849bba (127.20.240.65:37703)
I20260923 17:35:22.180250 28202 ts_manager.cc:194] Registered new tserver with Master: 4dbaafbf88034b6da6cfd705d4849bba (127.20.240.65:37703)
I20260923 17:35:22.264472 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 27798
I20260923 17:35:22.278383 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:35803
--local_ip_for_outbound_sockets=127.20.240.66
--tserver_master_addrs=127.20.240.126:36193
--webserver_port=41503
--webserver_interface=127.20.240.66
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.66.keytab
--principal=oryx/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.240.126:36193,127.20.240.125:43291
--tserver_master_addrs=127.20.240.126:36193,127.20.240.125:43291,127.20.240.124:33839 with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:35:22.379073 28352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: NON_VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: true } } }
I20260923 17:35:22.379509 28352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:22.381112 28583 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4: attempting to promote NON_VOTER 92f052fe126b402c805a99e40f9296c0 to VOTER
I20260923 17:35:22.381724 28583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } }
I20260923 17:35:22.383585 28212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 FOLLOWER]: Refusing update from remote peer 9dc5d7a24a374aefa3e191059db69da4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 17:35:22.383654 28402 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 1 LEARNER]: Refusing update from remote peer 9dc5d7a24a374aefa3e191059db69da4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 17:35:22.384230 28583 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 17:35:22.384634 28584 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 17:35:22.387430 28584 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 92f052fe126b402c805a99e40f9296c0 (127.20.240.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } } }
I20260923 17:35:22.388025 28212 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 92f052fe126b402c805a99e40f9296c0 (127.20.240.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } } }
I20260923 17:35:22.389262 28402 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 92f052fe126b402c805a99e40f9296c0 (127.20.240.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } } }
I20260923 17:35:22.392670 28430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } } }
I20260923 17:35:22.392932 28430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:35:22.395248 28584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } } }
I20260923 17:35:22.395514 28584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:22.395208 28352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } } }
I20260923 17:35:22.395639 28352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:22.395774 28584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } } }
I20260923 17:35:22.395977 28584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:22.396497 28587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9dc5d7a24a374aefa3e191059db69da4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } } }
I20260923 17:35:22.396875 28587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 17:35:22.607574 28582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:22.607877 28582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:22.608006 28582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:22.615300 28582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:22.615444 28582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:22.615619 28582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:35:22.626217 28582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.66.keytab
--principal=oryx/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:35803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=41503
--tserver_master_addrs=127.20.240.126:36193,127.20.240.125:43291,127.20.240.124:33839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:22.627620 28582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:22.628902 28582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:22.642270 28592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:22.642182 28593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:22.642194 28595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:22.643031 28582 server_base.cc:1034] running on GCE node
I20260923 17:35:22.643702 28582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:22.644712 28582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:22.645898 28582 hybrid_clock.cc:648] HybridClock initialized: now 1790184922645841 us; error 47 us; skew 500 ppm
I20260923 17:35:22.646173 28582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:22 dist-test-slave-ppjl krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184922, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:22.649245 28582 init.cc:377] Logged in from keytab as oryx/127.20.240.66@KRBTEST.COM (short username oryx)
I20260923 17:35:22.650687 28582 webserver.cc:460] Webserver started at http://127.20.240.66:41503/ using document root <none> and password file <none>
I20260923 17:35:22.651400 28582 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:22.651516 28582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:22.656800 28582 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260923 17:35:22.659588 28602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:22.661000 28582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:35:22.661149 28582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "841cba3f424b4c6ea31bb73a14f17314"
format_stamp: "Formatted at 2026-09-23 17:35:19 on dist-test-slave-ppjl"
server_key: "1463d2a2607e8d3ab27094f57de4b608"
server_key_iv: "7d40f4aa79082310045090b0c668244f"
server_key_version: "encryptionkey@0"
I20260923 17:35:22.662222 28582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:22.680981 28582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:22.685503 28582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:22.685760 28582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:22.686772 28582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:22.688247 28582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:22.688347 28582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:22.688459 28582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:22.688531 28582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:22.719084 28582 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:35803
I20260923 17:35:22.719151 28714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:35803 every 8 connection(s)
I20260923 17:35:22.720533 28582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:35:22.727982 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 28582
I20260923 17:35:22.728294 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 27931
Sep 23 17:35:22 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.240.67@KRBTEST.COM
Sep 23 17:35:22 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184922, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
Sep 23 17:35:22 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184922, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.125@KRBTEST.COM
Sep 23 17:35:22 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184922, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.124@KRBTEST.COM
I20260923 17:35:22.772025 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:22.729758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.20.240.67:45969)
0923 17:35:22.744800 (+ 15042us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:22.744811 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:22.744851 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:35:22.744947 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:22.744958 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:22.745344 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:22.748097 (+ 2753us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:22.751111 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.754128 (+ 3017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.756222 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.756971 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.757048 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:22.763551 (+ 6503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:22.763570 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:22.763588 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:22.763639 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:22.767825 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.768809 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.768819 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.768823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.768999 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.769689 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.769695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.769699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.770151 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:22.770536 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:22.771150 (+ 614us) server_negotiation.cc:299] Negotiation successful
0923 17:35:22.771495 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14782,"thread_start_us":14519,"threads_started":1}
I20260923 17:35:22.773607 28054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:22.778924 28719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:22.723487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:47389 (local address 127.20.240.126:36193)
0923 17:35:22.741545 (+ 18058us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:22.741552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:22.747895 (+ 6343us) server_negotiation.cc:407] Connection header received
0923 17:35:22.752902 (+ 5007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:22.752909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:22.753042 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:22.753164 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:22.756556 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.757332 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.760138 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.760786 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.760911 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:22.767370 (+ 6459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:22.767378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:22.767382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:22.767427 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:22.771241 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.774696 (+ 3455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.774704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.774708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.774934 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.776172 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.776180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.776184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.776614 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:22.776769 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:22.778165 (+ 1396us) server_negotiation.cc:299] Negotiation successful
0923 17:35:22.778471 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17883,"thread_start_us":18085,"threads_started":1}
I20260923 17:35:22.778964 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:22.724356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:44521 (local address 127.20.240.125:43291)
0923 17:35:22.750275 (+ 25919us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:22.750283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:22.754933 (+ 4650us) server_negotiation.cc:407] Connection header received
0923 17:35:22.755073 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:22.755079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:22.755186 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:22.755287 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:22.757668 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.758865 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.761029 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.761641 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.761808 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:22.766858 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:22.766866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:22.766871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:22.766931 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:22.770407 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.773310 (+ 2903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.773316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.773320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.773501 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.774636 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.774641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.774644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.775031 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:22.775228 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:22.778239 (+ 3011us) server_negotiation.cc:299] Negotiation successful
0923 17:35:22.778486 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25762,"thread_start_us":25943,"threads_started":1}
I20260923 17:35:22.780078 28720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:22.723532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:59667 (local address 127.20.240.124:33839)
0923 17:35:22.753688 (+ 30156us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:22.753694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:22.753714 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:35:22.753779 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:22.753785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:22.753957 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:22.754080 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:22.757648 (+ 3568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.758986 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.762033 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.762701 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.762826 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:22.768910 (+ 6084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:22.768918 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:22.768922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:22.768994 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:22.772738 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.773716 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.773723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.773728 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.773928 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:22.775552 (+ 1624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:22.775559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:22.775564 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:22.776000 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:22.777004 (+ 1004us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:22.779362 (+ 2358us) server_negotiation.cc:299] Negotiation successful
0923 17:35:22.779629 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30020,"thread_start_us":29892,"threads_started":1}
I20260923 17:35:22.780575 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:22.724546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:43291 (local address 127.20.240.66:44521)
0923 17:35:22.754827 (+ 30281us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:22.754842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:22.754948 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:22.755479 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:22.755484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:22.756114 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:22.757501 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:22.757512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.759052 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.759057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:22.760877 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:22.760891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.762122 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.762129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:22.762302 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:22.763345 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:22.763369 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:22.766619 (+ 3250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:22.771431 (+ 4812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:22.771441 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:22.771461 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:22.772311 (+ 850us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:22.774115 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:22.774122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:22.774127 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:22.774494 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:22.776833 (+ 2339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:22.776837 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:22.777480 (+ 643us) client_negotiation.cc:769] Sending connection context
0923 17:35:22.777827 (+ 347us) client_negotiation.cc:240] Negotiation successful
0923 17:35:22.778470 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30173,"mutex_wait_us":1171,"spinlock_wait_cycles":12288,"thread_start_us":29010,"threads_started":1}
I20260923 17:35:22.782269 28717 heartbeater.cc:347] Connected to a master server at 127.20.240.125:43291
I20260923 17:35:22.782589 28717 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:22.783476 28717 heartbeater.cc:511] Master 127.20.240.125:43291 requested a full tablet report, sending...
I20260923 17:35:22.784150 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:22.724245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:33839 (local address 127.20.240.66:59667)
0923 17:35:22.753493 (+ 29248us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:22.753513 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:22.753640 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:22.754309 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:22.754315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:22.755983 (+ 1668us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:22.757454 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:22.757469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.759140 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.759147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:22.761843 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:22.761858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.762975 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.762984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:22.764346 (+ 1362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:22.764953 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:22.764975 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:22.768668 (+ 3693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:22.772901 (+ 4233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:22.772906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:22.772910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:22.773512 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:22.775175 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:22.775182 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:22.775186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:22.775373 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:22.777065 (+ 1692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:22.777070 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:22.777480 (+ 410us) client_negotiation.cc:769] Sending connection context
0923 17:35:22.777827 (+ 347us) client_negotiation.cc:240] Negotiation successful
0923 17:35:22.783609 (+ 5782us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29075,"mutex_wait_us":5307,"spinlock_wait_cycles":87552,"thread_start_us":21415,"threads_started":1}
I20260923 17:35:22.785561 28202 ts_manager.cc:194] Registered new tserver with Master: 841cba3f424b4c6ea31bb73a14f17314 (127.20.240.66:35803)
I20260923 17:35:22.785638 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:22.724245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.66:47389)
0923 17:35:22.746419 (+ 22174us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:22.746494 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:22.752070 (+ 5576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:22.754083 (+ 2013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:22.754094 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:22.755235 (+ 1141us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:22.756357 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:22.756392 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.757491 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.757496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:22.759992 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:22.760008 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:22.761061 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:22.761070 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:22.761377 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:22.762164 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:22.762201 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:22.766126 (+ 3925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:22.773678 (+ 7552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:22.773683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:22.773687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:22.774118 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:22.775290 (+ 1172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:22.775296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:22.775300 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:22.775496 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:22.776808 (+ 1312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:22.776818 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:22.777527 (+ 709us) client_negotiation.cc:769] Sending connection context
0923 17:35:22.777823 (+ 296us) client_negotiation.cc:240] Negotiation successful
0923 17:35:22.785121 (+ 7298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21766,"mutex_wait_us":265,"spinlock_wait_cycles":54656,"thread_start_us":30017,"threads_started":1}
I20260923 17:35:22.785877 28715 heartbeater.cc:347] Connected to a master server at 127.20.240.124:33839
I20260923 17:35:22.786017 28715 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:22.786480 28715 heartbeater.cc:511] Master 127.20.240.124:33839 requested a full tablet report, sending...
I20260923 17:35:22.787830 28391 ts_manager.cc:194] Registered new tserver with Master: 841cba3f424b4c6ea31bb73a14f17314 (127.20.240.66:35803)
I20260923 17:35:22.787964 28716 heartbeater.cc:347] Connected to a master server at 127.20.240.126:36193
I20260923 17:35:22.788100 28716 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:22.788615 28716 heartbeater.cc:511] Master 127.20.240.126:36193 requested a full tablet report, sending...
I20260923 17:35:22.789701 27603 ts_manager.cc:194] Re-registered known tserver with Master: 841cba3f424b4c6ea31bb73a14f17314 (127.20.240.66:35803)
I20260923 17:35:22.790987 27603 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:47389
I20260923 17:35:22.859886 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 27931
I20260923 17:35:22.873689 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:45969
--local_ip_for_outbound_sockets=127.20.240.67
--tserver_master_addrs=127.20.240.126:36193
--webserver_port=33653
--webserver_interface=127.20.240.67
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.67.keytab
--principal=oryx/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.240.126:36193,127.20.240.125:43291
--tserver_master_addrs=127.20.240.126:36193,127.20.240.125:43291,127.20.240.124:33839 with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:35:23.177609 28571 heartbeater.cc:503] Master 127.20.240.126:36193 was elected leader, sending a full tablet report...
W20260923 17:35:23.204730 28728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:23.205024 28728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:23.205144 28728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:23.212352 28728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:23.212461 28728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:23.212628 28728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:35:23.222820 28728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.67.keytab
--principal=oryx/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:45969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=33653
--tserver_master_addrs=127.20.240.126:36193,127.20.240.125:43291,127.20.240.124:33839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:23.224192 28728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:23.225528 28728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:23.238714 28734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:23.238782 28735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:23.238754 28737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:23.239563 28728 server_base.cc:1034] running on GCE node
I20260923 17:35:23.240244 28728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:23.241189 28728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:23.242391 28728 hybrid_clock.cc:648] HybridClock initialized: now 1790184923242338 us; error 45 us; skew 500 ppm
I20260923 17:35:23.242681 28728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:23 dist-test-slave-ppjl krb5kdc[27558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184923, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:23.245904 28728 init.cc:377] Logged in from keytab as oryx/127.20.240.67@KRBTEST.COM (short username oryx)
I20260923 17:35:23.247318 28728 webserver.cc:460] Webserver started at http://127.20.240.67:33653/ using document root <none> and password file <none>
I20260923 17:35:23.247973 28728 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:23.248106 28728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:23.253448 28728 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260923 17:35:23.256165 28744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:23.257565 28728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:35:23.257740 28728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "494adc561a604d76954288bed8325b03"
format_stamp: "Formatted at 2026-09-23 17:35:19 on dist-test-slave-ppjl"
server_key: "b15b992e2b45d8465a83e3b4f36e8b51"
server_key_iv: "2873f86b768ede6eeaab5be8195e50e3"
server_key_version: "encryptionkey@0"
I20260923 17:35:23.258754 28728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:23.283907 28728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:23.288981 28728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:23.289286 28728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:23.290359 28728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:23.291872 28728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:23.291975 28728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:23.292080 28728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:23.292148 28728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:23.321830 28728 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:45969
I20260923 17:35:23.321892 28856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:45969 every 8 connection(s)
I20260923 17:35:23.323348 28728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:35:23.323772 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 28728
Sep 23 17:35:23 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184923, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.125@KRBTEST.COM
I20260923 17:35:23.361744 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.328001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:50691 (local address 127.20.240.125:43291)
0923 17:35:23.338267 (+ 10266us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.338274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.338296 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:35:23.338365 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.338371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.338454 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.338570 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:23.342250 (+ 3680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.343367 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.345638 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.346330 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.346459 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.352517 (+ 6058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:23.352525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:23.352531 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:23.352604 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:23.356722 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.358060 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.358068 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.358072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.358291 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.358907 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.358913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.358917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.359338 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:23.359499 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.360691 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.360977 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10126,"thread_start_us":10009,"threads_started":1}
I20260923 17:35:23.365248 28863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.327313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:43291 (local address 127.20.240.67:50691)
0923 17:35:23.333963 (+ 6650us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:23.334023 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:23.336778 (+ 2755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:23.340211 (+ 3433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:23.340214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:23.340722 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:23.342054 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.342065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.343585 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.343591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.345492 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.345504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.346775 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.346783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.347116 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.347993 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:23.348027 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:23.352167 (+ 4140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:23.357038 (+ 4871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:23.357051 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:23.357077 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:23.357880 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:23.358461 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:23.358467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:23.358471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:23.358770 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:23.359616 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:23.359627 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:23.360172 (+ 545us) client_negotiation.cc:769] Sending connection context
0923 17:35:23.362500 (+ 2328us) client_negotiation.cc:240] Negotiation successful
0923 17:35:23.363550 (+ 1050us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6300,"mutex_wait_us":698,"spinlock_wait_cycles":53888,"thread_start_us":6044,"threads_started":1}
I20260923 17:35:23.367503 28859 heartbeater.cc:347] Connected to a master server at 127.20.240.125:43291
I20260923 17:35:23.367895 28859 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:23.369041 28859 heartbeater.cc:511] Master 127.20.240.125:43291 requested a full tablet report, sending...
Sep 23 17:35:23 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184923, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
Sep 23 17:35:23 dist-test-slave-ppjl krb5kdc[27558](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184923, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.124@KRBTEST.COM
I20260923 17:35:23.371503 28202 ts_manager.cc:194] Registered new tserver with Master: 494adc561a604d76954288bed8325b03 (127.20.240.67:45969)
I20260923 17:35:23.386416 28864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.327858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.67:37693)
0923 17:35:23.337656 (+ 9798us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:23.337684 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:23.337810 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:23.339482 (+ 1672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:23.339494 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:23.340763 (+ 1269us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:23.341874 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.341902 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.344024 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.344031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.345827 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.345843 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.362687 (+ 16844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.362694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.362876 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.366139 (+ 3263us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:23.366162 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:23.375331 (+ 9169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:23.382598 (+ 7267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:23.382603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:23.382607 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:23.383304 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:23.384053 (+ 749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:23.384060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:23.384064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:23.384249 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:23.385235 (+ 986us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:23.385240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:23.385429 (+ 189us) client_negotiation.cc:769] Sending connection context
0923 17:35:23.385575 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 17:35:23.385897 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9663,"mutex_wait_us":331,"spinlock_wait_cycles":6912,"thread_start_us":9492,"threads_started":1}
I20260923 17:35:23.386463 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.326352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:37693 (local address 127.20.240.126:36193)
0923 17:35:23.338476 (+ 12124us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.338483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.338510 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:35:23.338590 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.338596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.338723 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.338853 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:23.342282 (+ 3429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.343381 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.361466 (+ 18085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.362171 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.362277 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.376110 (+ 13833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:23.376125 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:23.376130 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:23.376212 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:23.381516 (+ 5304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.383453 (+ 1937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.383461 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.383465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.383677 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.384645 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.384651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.384654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.385077 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:23.385261 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.385740 (+ 479us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.386019 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11984,"thread_start_us":11878,"threads_started":1}
I20260923 17:35:23.388273 28858 heartbeater.cc:347] Connected to a master server at 127.20.240.126:36193
I20260923 17:35:23.388455 28858 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:23.389114 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.327430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:39423 (local address 127.20.240.124:33839)
0923 17:35:23.333301 (+ 5871us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.333309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.339393 (+ 6084us) server_negotiation.cc:407] Connection header received
0923 17:35:23.339464 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.339469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.339559 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.339657 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:23.342081 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.343312 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.359477 (+ 16165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.360188 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.360298 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.377580 (+ 17282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:23.377588 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:23.377594 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:23.377645 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:23.381515 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.382628 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.382636 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.382640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.382852 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.383887 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.383893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.383897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.384403 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:23.384554 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.388456 (+ 3902us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.388739 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5733,"thread_start_us":5617,"threads_started":1}
I20260923 17:35:23.389259 28858 heartbeater.cc:511] Master 127.20.240.126:36193 requested a full tablet report, sending...
I20260923 17:35:23.389956 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.327127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:33839 (local address 127.20.240.67:39423)
0923 17:35:23.339109 (+ 11982us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:23.339136 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:23.339273 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:23.339845 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:23.339849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:23.340645 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:23.341871 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.341901 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.344048 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.344053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.357399 (+ 13346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.357423 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.360701 (+ 3278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.360709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.360910 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.365509 (+ 4599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:23.365536 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:23.375802 (+ 10266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:23.381729 (+ 5927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:23.381735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:23.381740 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:23.382444 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:23.383018 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:23.383024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:23.383028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:23.383302 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:23.385353 (+ 2051us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:23.385359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:23.388087 (+ 2728us) client_negotiation.cc:769] Sending connection context
0923 17:35:23.389125 (+ 1038us) client_negotiation.cc:240] Negotiation successful
0923 17:35:23.389469 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9783,"mutex_wait_us":91,"spinlock_wait_cycles":9472,"thread_start_us":9648,"threads_started":1}
I20260923 17:35:23.390463 27603 ts_manager.cc:194] Re-registered known tserver with Master: 494adc561a604d76954288bed8325b03 (127.20.240.67:45969)
I20260923 17:35:23.390897 28857 heartbeater.cc:347] Connected to a master server at 127.20.240.124:33839
I20260923 17:35:23.391033 28857 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:23.391551 28857 heartbeater.cc:511] Master 127.20.240.124:33839 requested a full tablet report, sending...
I20260923 17:35:23.391762 27603 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:37693
I20260923 17:35:23.392860 28391 ts_manager.cc:194] Registered new tserver with Master: 494adc561a604d76954288bed8325b03 (127.20.240.67:45969)
I20260923 17:35:23.403264 28872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.343691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47562 (local address 127.20.240.126:36193)
0923 17:35:23.344159 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.344165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.364820 (+ 20655us) server_negotiation.cc:407] Connection header received
0923 17:35:23.370480 (+ 5660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.370486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.370600 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.370693 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:23.374710 (+ 4017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.376250 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.386836 (+ 10586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.387598 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.387716 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.394849 (+ 7133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:23.394854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:23.394858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:23.394890 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:23.398801 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.399929 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.399935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.399939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.400077 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.400482 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.400487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.400491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.400808 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:23.401067 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.401461 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.401701 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":209,"threads_started":1}
I20260923 17:35:23.409147 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.346311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.20.240.125:43291)
0923 17:35:23.362424 (+ 16113us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.362430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.387393 (+ 24963us) server_negotiation.cc:407] Connection header received
0923 17:35:23.387678 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.387684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.387789 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.387863 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:23.393375 (+ 5512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.394527 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.396877 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.397581 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.397732 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.401377 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:23.401385 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:23.401390 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:23.401434 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:23.404979 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.405670 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.405676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.405680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.405826 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.407293 (+ 1467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.407299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.407301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.407634 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:23.407925 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.408435 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.408859 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15994,"thread_start_us":21651,"threads_started":1}
I20260923 17:35:23.410772 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.346307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41318 (local address 127.20.240.124:33839)
0923 17:35:23.383963 (+ 37656us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.383969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.386700 (+ 2731us) server_negotiation.cc:407] Connection header received
0923 17:35:23.386763 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.386768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.386853 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.386950 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:23.393503 (+ 6553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.394712 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.397948 (+ 3236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.398678 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.398803 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.402273 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:23.402280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:23.402285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:23.402329 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:23.405817 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.406955 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.406960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.406962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.407148 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.408333 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.408338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.408341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.408705 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:23.408940 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.410102 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.410353 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37493,"thread_start_us":37303,"threads_started":1}
I20260923 17:35:23.408330 27603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:35:23.417186 27603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:35:23.459318 28887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.447724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:45969 (local address 127.0.0.1:39050)
0923 17:35:23.449950 (+ 2226us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:23.449970 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:23.450199 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:23.451200 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:23.451205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:23.451231 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:23.451598 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.451607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.453784 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.453790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.456308 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.456322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.458329 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.458336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.458557 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.458594 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 17:35:23.458689 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 17:35:23.458827 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2097,"mutex_wait_us":1,"thread_start_us":1880,"threads_started":1}
I20260923 17:35:23.459353 28888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.448020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.20.240.67:45969)
0923 17:35:23.449375 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.449385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.450066 (+ 681us) server_negotiation.cc:407] Connection header received
0923 17:35:23.450421 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.450431 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.450708 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.450868 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:23.451797 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.453610 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.456467 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.458105 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.458237 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.458456 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.458764 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.458940 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1132,"thread_start_us":432,"threads_started":1}
I20260923 17:35:23.460314 28889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.449867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:35803 (local address 127.0.0.1:35268)
0923 17:35:23.450445 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:23.450461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:23.450573 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:23.452766 (+ 2193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:23.452771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:23.452796 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:23.453364 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.453376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.454901 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.454906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.457411 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.457426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.459536 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.459543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.459724 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.459747 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:35:23.459829 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:35:23.459967 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":2,"thread_start_us":320,"threads_started":1}
I20260923 17:35:23.460939 28890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.450167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.20.240.66:35803)
0923 17:35:23.452143 (+ 1976us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.452151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.452181 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:35:23.452288 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.452295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.452527 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.453599 (+ 1072us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:23.453796 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.454774 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.457604 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.458974 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.459086 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.459305 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.460539 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.460699 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1815,"thread_start_us":1729,"threads_started":1}
I20260923 17:35:23.464701 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.451090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:37703 (local address 127.0.0.1:36500)
0923 17:35:23.455683 (+ 4593us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:23.455711 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:23.455879 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:23.456866 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:23.456870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:23.456897 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:23.457396 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.457408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.459235 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.459240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.461427 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.461443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.463792 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.463800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.463995 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.464048 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 17:35:23.464137 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 17:35:23.464273 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4454,"thread_start_us":4243,"threads_started":1}
I20260923 17:35:23.464696 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.450789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.20.240.65:37703)
0923 17:35:23.451383 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.451393 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.455856 (+ 4463us) server_negotiation.cc:407] Connection header received
0923 17:35:23.456033 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.456041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.456307 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.456546 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:23.457736 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.459048 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.461588 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.463243 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.463383 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.463597 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.464194 (+ 597us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.464381 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":534,"threads_started":1}
I20260923 17:35:23.464427 28792 tablet_service.cc:1467] Processing CreateTablet for tablet dff9fce5751f49a4a3a7b3ab4360e99c (DEFAULT_TABLE table=test-table [id=6aa94c3f74c9422d80aaf24ed8675919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:23.464720 28650 tablet_service.cc:1467] Processing CreateTablet for tablet dff9fce5751f49a4a3a7b3ab4360e99c (DEFAULT_TABLE table=test-table [id=6aa94c3f74c9422d80aaf24ed8675919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:23.465901 28792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff9fce5751f49a4a3a7b3ab4360e99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:23.466423 28650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff9fce5751f49a4a3a7b3ab4360e99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:23.468852 28505 tablet_service.cc:1467] Processing CreateTablet for tablet dff9fce5751f49a4a3a7b3ab4360e99c (DEFAULT_TABLE table=test-table [id=6aa94c3f74c9422d80aaf24ed8675919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:23.470327 28505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff9fce5751f49a4a3a7b3ab4360e99c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:23.481103 28893 tablet_bootstrap.cc:492] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314: Bootstrap starting.
I20260923 17:35:23.484619 28893 tablet_bootstrap.cc:654] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:23.485845 28893 log.cc:824] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:23.487071 28894 tablet_bootstrap.cc:492] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03: Bootstrap starting.
I20260923 17:35:23.491953 28894 tablet_bootstrap.cc:654] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:23.492960 28896 tablet_bootstrap.cc:492] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba: Bootstrap starting.
I20260923 17:35:23.493547 28894 log.cc:824] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:23.494401 28893 tablet_bootstrap.cc:492] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314: No bootstrap required, opened a new log
I20260923 17:35:23.494745 28893 ts_tablet_manager.cc:1397] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260923 17:35:23.496587 28896 tablet_bootstrap.cc:654] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:23.497298 28894 tablet_bootstrap.cc:492] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03: No bootstrap required, opened a new log
I20260923 17:35:23.497596 28894 ts_tablet_manager.cc:1397] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 17:35:23.497849 28896 log.cc:824] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:23.504444 28896 tablet_bootstrap.cc:492] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba: No bootstrap required, opened a new log
I20260923 17:35:23.504778 28896 ts_tablet_manager.cc:1397] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260923 17:35:23.506531 28893 raft_consensus.cc:348] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } }
I20260923 17:35:23.507026 28893 raft_consensus.cc:374] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:23.507180 28893 raft_consensus.cc:729] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841cba3f424b4c6ea31bb73a14f17314, State: Initialized, Role: FOLLOWER
I20260923 17:35:23.507894 28893 consensus_queue.cc:260] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } }
I20260923 17:35:23.509169 28716 heartbeater.cc:503] Master 127.20.240.126:36193 was elected leader, sending a full tablet report...
I20260923 17:35:23.509980 28893 ts_tablet_manager.cc:1428] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314: Time spent starting tablet: real 0.015s user 0.012s sys 0.005s
I20260923 17:35:23.510676 28894 raft_consensus.cc:348] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } }
I20260923 17:35:23.511204 28894 raft_consensus.cc:374] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:23.511329 28894 raft_consensus.cc:729] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494adc561a604d76954288bed8325b03, State: Initialized, Role: FOLLOWER
I20260923 17:35:23.512008 28894 consensus_queue.cc:260] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } }
I20260923 17:35:23.515059 28896 raft_consensus.cc:348] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } }
I20260923 17:35:23.515583 28896 raft_consensus.cc:374] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:23.515687 28896 raft_consensus.cc:729] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbaafbf88034b6da6cfd705d4849bba, State: Initialized, Role: FOLLOWER
I20260923 17:35:23.516412 28896 consensus_queue.cc:260] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } }
I20260923 17:35:23.521044 28896 ts_tablet_manager.cc:1428] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260923 17:35:23.521638 28894 ts_tablet_manager.cc:1428] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03: Time spent starting tablet: real 0.024s user 0.019s sys 0.004s
I20260923 17:35:23.521919 28858 heartbeater.cc:503] Master 127.20.240.126:36193 was elected leader, sending a full tablet report...
W20260923 17:35:23.590080 28867 tablet.cc:2369] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:35:23.645120 28573 tablet.cc:2369] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:35:23.734040 28901 raft_consensus.cc:482] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:35:23.734365 28901 raft_consensus.cc:504] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } }
I20260923 17:35:23.736230 28901 leader_election.cc:288] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 494adc561a604d76954288bed8325b03 (127.20.240.67:45969), 841cba3f424b4c6ea31bb73a14f17314 (127.20.240.66:35803)
I20260923 17:35:23.738937 28899 raft_consensus.cc:482] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:35:23.739275 28899 raft_consensus.cc:504] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } }
W20260923 17:35:23.741369 28718 tablet.cc:2369] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:35:23.741427 28899 leader_election.cc:288] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 494adc561a604d76954288bed8325b03 (127.20.240.67:45969), 4dbaafbf88034b6da6cfd705d4849bba (127.20.240.65:37703)
I20260923 17:35:23.744779 28888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.736608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:37639 (local address 127.20.240.67:45969)
0923 17:35:23.736895 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.736903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.737423 (+ 520us) server_negotiation.cc:407] Connection header received
0923 17:35:23.737672 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.737679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.737764 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.737854 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:23.738597 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.739629 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.742136 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.743523 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.743655 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.743728 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.744376 (+ 648us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.744514 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 17:35:23.745748 28890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.737471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:44677 (local address 127.20.240.66:35803)
0923 17:35:23.737703 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.737710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.738264 (+ 554us) server_negotiation.cc:407] Connection header received
0923 17:35:23.738402 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.738408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.738482 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.738563 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:23.739289 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.740506 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.743138 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.744508 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.744623 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.744699 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.745350 (+ 651us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.745474 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:35:23.745765 28902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.736477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:45969 (local address 127.20.240.65:37639)
0923 17:35:23.737268 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:23.737285 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:23.737488 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:23.737923 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:23.737928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:23.737980 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:23.738436 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.738449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.739809 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.739812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.741967 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.741982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.743919 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.743926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.744110 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.744145 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:35:23.745240 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0923 17:35:23.745375 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":424,"threads_started":1}
I20260923 17:35:23.746479 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.737285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:35803 (local address 127.20.240.65:44677)
0923 17:35:23.738132 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:23.738149 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:23.738284 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:23.738592 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:23.738596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:23.738621 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:23.739021 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.739033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.740682 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.740688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.742944 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.742960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.744993 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.744999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.745173 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.745205 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:35:23.745980 (+ 775us) client_negotiation.cc:240] Negotiation successful
0923 17:35:23.746108 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":562,"threads_started":1}
I20260923 17:35:23.747460 28812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff9fce5751f49a4a3a7b3ab4360e99c" candidate_uuid: "4dbaafbf88034b6da6cfd705d4849bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494adc561a604d76954288bed8325b03" is_pre_election: true
I20260923 17:35:23.747532 28670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff9fce5751f49a4a3a7b3ab4360e99c" candidate_uuid: "4dbaafbf88034b6da6cfd705d4849bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841cba3f424b4c6ea31bb73a14f17314" is_pre_election: true
I20260923 17:35:23.748088 28670 raft_consensus.cc:2454] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbaafbf88034b6da6cfd705d4849bba in term 0.
I20260923 17:35:23.748149 28812 raft_consensus.cc:2454] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbaafbf88034b6da6cfd705d4849bba in term 0.
I20260923 17:35:23.748873 28458 leader_election.cc:302] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dbaafbf88034b6da6cfd705d4849bba, 841cba3f424b4c6ea31bb73a14f17314; no voters:
I20260923 17:35:23.749555 28901 raft_consensus.cc:2781] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:35:23.749760 28901 raft_consensus.cc:482] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:35:23.749897 28901 raft_consensus.cc:3037] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:23.753429 28901 raft_consensus.cc:504] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } }
I20260923 17:35:23.754197 28901 leader_election.cc:288] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [CANDIDATE]: Term 1 election: Requested vote from peers 494adc561a604d76954288bed8325b03 (127.20.240.67:45969), 841cba3f424b4c6ea31bb73a14f17314 (127.20.240.66:35803)
I20260923 17:35:23.754802 28670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff9fce5751f49a4a3a7b3ab4360e99c" candidate_uuid: "4dbaafbf88034b6da6cfd705d4849bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841cba3f424b4c6ea31bb73a14f17314"
I20260923 17:35:23.754799 28812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff9fce5751f49a4a3a7b3ab4360e99c" candidate_uuid: "4dbaafbf88034b6da6cfd705d4849bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494adc561a604d76954288bed8325b03"
I20260923 17:35:23.755034 28670 raft_consensus.cc:3037] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:23.755023 28812 raft_consensus.cc:3037] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:23.755424 28888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.742289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:56957 (local address 127.20.240.67:45969)
0923 17:35:23.745202 (+ 2913us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.745207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.747616 (+ 2409us) server_negotiation.cc:407] Connection header received
0923 17:35:23.747833 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.747838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.747908 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.747966 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:23.749286 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.750367 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.753014 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.754016 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.754179 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.754256 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.754845 (+ 589us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.754972 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2801,"thread_start_us":4148,"threads_started":1}
I20260923 17:35:23.755486 28904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.741833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:45969 (local address 127.20.240.66:56957)
0923 17:35:23.747453 (+ 5620us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:23.747477 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:23.747666 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:23.748408 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:23.748414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:23.748464 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:23.749111 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.749126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.750510 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.750516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.752832 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.752847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.754453 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.754460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.754646 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.754700 (+ 54us) client_negotiation.cc:769] Sending connection context
0923 17:35:23.754803 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 17:35:23.754952 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5485,"mutex_wait_us":24,"thread_start_us":5864,"threads_started":1}
I20260923 17:35:23.756668 28811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff9fce5751f49a4a3a7b3ab4360e99c" candidate_uuid: "841cba3f424b4c6ea31bb73a14f17314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494adc561a604d76954288bed8325b03" is_pre_election: true
I20260923 17:35:23.757040 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.748689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:35871 (local address 127.20.240.65:37703)
0923 17:35:23.749260 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.749266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.749282 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:35:23.749470 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.749475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.749553 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.749643 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:23.750228 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.751091 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.753088 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.754315 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.754460 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.754525 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.756613 (+ 2088us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.756756 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423}
I20260923 17:35:23.757397 28906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.748183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:37703 (local address 127.20.240.66:35871)
0923 17:35:23.749188 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:23.749207 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:23.749343 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:23.749670 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:23.749675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:23.749701 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:23.750071 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.750085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.751234 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.751239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.752868 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:23.752882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.756245 (+ 3363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.756254 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:23.756451 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.756476 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:35:23.756566 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 17:35:23.756704 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":3,"thread_start_us":718,"threads_started":1}
I20260923 17:35:23.758352 28525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff9fce5751f49a4a3a7b3ab4360e99c" candidate_uuid: "841cba3f424b4c6ea31bb73a14f17314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dbaafbf88034b6da6cfd705d4849bba" is_pre_election: true
I20260923 17:35:23.758738 28670 raft_consensus.cc:2454] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbaafbf88034b6da6cfd705d4849bba in term 1.
I20260923 17:35:23.758884 28525 raft_consensus.cc:2379] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 841cba3f424b4c6ea31bb73a14f17314 in current term 1: Already voted for candidate 4dbaafbf88034b6da6cfd705d4849bba in this term.
I20260923 17:35:23.759231 28458 leader_election.cc:302] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dbaafbf88034b6da6cfd705d4849bba, 841cba3f424b4c6ea31bb73a14f17314; no voters:
I20260923 17:35:23.759658 28901 raft_consensus.cc:2781] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:23.759661 28603 leader_election.cc:302] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 841cba3f424b4c6ea31bb73a14f17314; no voters: 494adc561a604d76954288bed8325b03, 4dbaafbf88034b6da6cfd705d4849bba
I20260923 17:35:23.759824 28812 raft_consensus.cc:2454] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbaafbf88034b6da6cfd705d4849bba in term 1.
I20260923 17:35:23.760085 28899 raft_consensus.cc:2726] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 17:35:23.760456 28901 raft_consensus.cc:686] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [term 1 LEADER]: Becoming Leader. State: Replica: 4dbaafbf88034b6da6cfd705d4849bba, State: Running, Role: LEADER
I20260923 17:35:23.761065 28901 consensus_queue.cc:237] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } }
I20260923 17:35:23.765942 27603 catalog_manager.cc:5495] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dbaafbf88034b6da6cfd705d4849bba (127.20.240.65). New cstate: current_term: 1 leader_uuid: "4dbaafbf88034b6da6cfd705d4849bba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } health_report { overall_health: HEALTHY } } }
I20260923 17:35:23.829420 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.823352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.20.240.65:37703)
0923 17:35:23.823530 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.823535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.823554 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:35:23.823690 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.823693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.823772 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.823903 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:35:23.824422 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.825404 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.826943 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.827590 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.827724 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.828244 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:35:23.828786 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:35:23.828924 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.829079 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.829153 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:35:23.838780 28812 raft_consensus.cc:1261] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03 [term 1 FOLLOWER]: Refusing update from remote peer 4dbaafbf88034b6da6cfd705d4849bba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:23.838840 28670 raft_consensus.cc:1261] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314 [term 1 FOLLOWER]: Refusing update from remote peer 4dbaafbf88034b6da6cfd705d4849bba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:23.839840 28907 consensus_queue.cc:1035] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [LEADER]: Connected to new peer: Peer: permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:23.840435 28901 consensus_queue.cc:1035] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [LEADER]: Connected to new peer: Peer: permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:23.861204 28911 mvcc.cc:205] Tried to move back new op lower bound from 7332597448032034816 to 7332597447729704960. Current Snapshot: MvccSnapshot[applied={T|T < 7332597448032034816}]
I20260923 17:35:23.872951 28913 mvcc.cc:205] Tried to move back new op lower bound from 7332597448032034816 to 7332597447729704960. Current Snapshot: MvccSnapshot[applied={T|T < 7332597448032034816}]
I20260923 17:35:23.881240 27603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:23.881502 27603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:23.889010 27603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dc5d7a24a374aefa3e191059db69da4: Sending DeleteTablet for 3 replicas of tablet dff9fce5751f49a4a3a7b3ab4360e99c
I20260923 17:35:23.890077 28650 tablet_service.cc:1514] Processing DeleteTablet for tablet dff9fce5751f49a4a3a7b3ab4360e99c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:23 UTC) from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:35268
I20260923 17:35:23.890208 28792 tablet_service.cc:1514] Processing DeleteTablet for tablet dff9fce5751f49a4a3a7b3ab4360e99c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:23 UTC) from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:39050
I20260923 17:35:23.890236 28505 tablet_service.cc:1514] Processing DeleteTablet for tablet dff9fce5751f49a4a3a7b3ab4360e99c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:23 UTC) from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:36500
I20260923 17:35:23.900911 28921 tablet_replica.cc:330] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba: stopping tablet replica
I20260923 17:35:23.901465 28921 raft_consensus.cc:2229] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:35:23.902163 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 27571
I20260923 17:35:23.902283 28921 raft_consensus.cc:2258] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:23.903203 28922 tablet_replica.cc:330] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03: stopping tablet replica
I20260923 17:35:23.903515 27633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:23.903719 28922 raft_consensus.cc:2229] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:23.904397 28922 raft_consensus.cc:2258] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:23.906545 28922 ts_tablet_manager.cc:1905] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:23.914650 28920 tablet_replica.cc:330] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314: stopping tablet replica
I20260923 17:35:23.915271 28922 ts_tablet_manager.cc:1918] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:23.915282 28920 raft_consensus.cc:2229] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:23.915488 28922 log.cc:1196] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/dff9fce5751f49a4a3a7b3ab4360e99c
I20260923 17:35:23.915998 28920 raft_consensus.cc:2258] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:23.916095 28922 ts_tablet_manager.cc:1939] T dff9fce5751f49a4a3a7b3ab4360e99c P 494adc561a604d76954288bed8325b03: Deleting consensus metadata
I20260923 17:35:23.918345 28920 ts_tablet_manager.cc:1905] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:23.920384 28921 ts_tablet_manager.cc:1905] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:23.929451 28921 ts_tablet_manager.cc:1918] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:23.929673 28920 ts_tablet_manager.cc:1918] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:23.929745 28921 log.cc:1196] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/dff9fce5751f49a4a3a7b3ab4360e99c
I20260923 17:35:23.929908 28920 log.cc:1196] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/dff9fce5751f49a4a3a7b3ab4360e99c
I20260923 17:35:23.930704 28920 ts_tablet_manager.cc:1939] T dff9fce5751f49a4a3a7b3ab4360e99c P 841cba3f424b4c6ea31bb73a14f17314: Deleting consensus metadata
I20260923 17:35:23.930804 28921 ts_tablet_manager.cc:1939] T dff9fce5751f49a4a3a7b3ab4360e99c P 4dbaafbf88034b6da6cfd705d4849bba: Deleting consensus metadata
I20260923 17:35:23.933245 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.893647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53204 (local address 127.20.240.125:43291)
0923 17:35:23.893835 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.893841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.900119 (+ 6278us) server_negotiation.cc:407] Connection header received
0923 17:35:23.901130 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.901137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.901234 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.901343 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:23.915394 (+ 14051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.916537 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.918460 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.919131 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.919263 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.923445 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:23.923452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:23.923457 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:23.923511 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:23.927719 (+ 4208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.929028 (+ 1309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.929034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.929037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.929305 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.930000 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.930005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.930008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.930654 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:23.930944 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.932518 (+ 1574us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.932795 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:35:23.951695 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:23.900716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.20.240.124:33839)
0923 17:35:23.900880 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:23.900886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:23.932345 (+ 31459us) server_negotiation.cc:407] Connection header received
0923 17:35:23.934499 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:23.934506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:23.934644 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:23.934749 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:23.936853 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.938045 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.940069 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:23.940692 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:23.940822 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:23.944507 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:23.944515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:23.944520 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:23.944589 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:23.948368 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.949293 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.949299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.949303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.949483 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:23.949929 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:23.949934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:23.949938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:23.950324 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:23.950568 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:23.951021 (+ 453us) server_negotiation.cc:299] Negotiation successful
0923 17:35:23.951310 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:35:24.054193 27587 catalog_manager.cc:4841] TS 494adc561a604d76954288bed8325b03 (127.20.240.67:45969): tablet dff9fce5751f49a4a3a7b3ab4360e99c (table test-table [id=6aa94c3f74c9422d80aaf24ed8675919]) successfully deleted
W20260923 17:35:24.054503 27587 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.20.240.67:45969, user_credentials={real_user=slave}} blocked reactor thread for 136560us
I20260923 17:35:24.054805 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 27571
W20260923 17:35:24.073268 28606 connection.cc:535] server connection from 127.0.0.1:35268 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 17:35:24.109488 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:24.086138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53218 (local address 127.20.240.125:43291)
0923 17:35:24.086318 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:24.086324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:24.093837 (+ 7513us) server_negotiation.cc:407] Connection header received
0923 17:35:24.094047 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:24.094052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:24.094139 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:24.094232 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:24.096688 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:24.097459 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:24.099583 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:24.100019 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:24.100135 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:24.103973 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:24.103979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:24.103982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:24.104017 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:24.106550 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:24.107302 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:24.107308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:24.107311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:24.107446 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:24.107859 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:24.107863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:24.107866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:24.108215 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:24.108455 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:24.108884 (+ 429us) server_negotiation.cc:299] Negotiation successful
0923 17:35:24.109140 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:35:24.115573 28874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:24.085733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41332 (local address 127.20.240.124:33839)
0923 17:35:24.085963 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:24.085969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:24.100929 (+ 14960us) server_negotiation.cc:407] Connection header received
0923 17:35:24.101081 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:24.101087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:24.101174 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:24.101256 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:24.103656 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:24.104770 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:24.106598 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:24.107088 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:24.107233 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:24.110326 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:24.110332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:24.110335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:24.110371 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:24.113017 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:24.113666 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:24.113670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:24.113672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:24.113799 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:24.114157 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:24.114161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:24.114163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:24.114448 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:24.114676 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:24.115008 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 17:35:24.115275 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:35:24.872195 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:24.871212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.67:55565)
0923 17:35:24.871740 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:24.871946 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":286,"threads_started":1}
W20260923 17:35:24.872649 28858 heartbeater.cc:644] Failed to heartbeat to 127.20.240.126:36193 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
I20260923 17:35:24.875389 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:24.874578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.65:57655)
0923 17:35:24.875032 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:24.875210 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":233,"threads_started":1}
W20260923 17:35:24.875735 28571 heartbeater.cc:644] Failed to heartbeat to 127.20.240.126:36193 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
I20260923 17:35:24.879561 28934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:24.878812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.66:42245)
0923 17:35:24.879284 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:24.879402 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":239,"threads_started":1}
W20260923 17:35:24.880015 28716 heartbeater.cc:644] Failed to heartbeat to 127.20.240.126:36193 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
I20260923 17:35:25.388988 28935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dc5d7a24a374aefa3e191059db69da4)
I20260923 17:35:25.389266 28935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } }
I20260923 17:35:25.389372 28936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9dc5d7a24a374aefa3e191059db69da4)
I20260923 17:35:25.389605 28936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } }
I20260923 17:35:25.390106 28935 leader_election.cc:288] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dc5d7a24a374aefa3e191059db69da4 (127.20.240.126:36193), 90f6ec2f0d9740be81aa4929afc03075 (127.20.240.125:43291)
I20260923 17:35:25.391254 28936 leader_election.cc:288] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9dc5d7a24a374aefa3e191059db69da4 (127.20.240.126:36193), 92f052fe126b402c805a99e40f9296c0 (127.20.240.124:33839)
I20260923 17:35:25.398605 28937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:25.390351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:47996)
0923 17:35:25.398109 (+ 7758us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:25.398309 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7614,"thread_start_us":7466,"threads_started":1}
W20260923 17:35:25.399009 28377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
I20260923 17:35:25.401046 28938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:25.400399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:48008)
0923 17:35:25.400599 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:25.400813 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260923 17:35:25.401325 28377 leader_election.cc:334] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dc5d7a24a374aefa3e191059db69da4 (127.20.240.126:36193): Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
I20260923 17:35:25.403801 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:25.400268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:48004)
0923 17:35:25.403239 (+ 2971us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:25.403419 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2818,"thread_start_us":1319,"threads_started":1}
W20260923 17:35:25.404132 28188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
I20260923 17:35:25.405931 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:25.405542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:48032)
0923 17:35:25.405710 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:25.405793 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 17:35:25.406275 28188 leader_election.cc:334] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9dc5d7a24a374aefa3e191059db69da4 (127.20.240.126:36193): Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
I20260923 17:35:25.408145 28940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:25.391672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:33839 (local address 127.0.0.1:41338)
0923 17:35:25.398443 (+ 6771us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:25.398463 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:25.398623 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:25.402036 (+ 3413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:25.402041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:25.402074 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:25.402457 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:25.402470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:25.404107 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:25.404112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:25.406104 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:25.406118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:25.407348 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:25.407354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:25.407538 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:25.407562 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:35:25.407641 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 17:35:25.407784 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6637,"mutex_wait_us":25,"thread_start_us":7372,"threads_started":1}
I20260923 17:35:25.408205 28943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:25.400752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.20.240.124:33839)
0923 17:35:25.401668 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:25.401707 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:25.401738 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:35:25.401796 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:25.401800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:25.401867 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:25.402169 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:25.402781 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:25.403975 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:25.406249 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:25.407185 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:25.407345 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:25.407419 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:25.407678 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 17:35:25.407804 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":799,"thread_start_us":603,"threads_started":1}
I20260923 17:35:25.408614 28937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:25.390369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:43291 (local address 127.0.0.1:53226)
0923 17:35:25.398853 (+ 8484us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:25.398870 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:25.398995 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:25.401101 (+ 2106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:25.401106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:25.401683 (+ 577us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:25.401939 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:25.401951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:25.405067 (+ 3116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:25.405072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:25.406701 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:25.406714 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:25.407879 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:25.407885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:25.408063 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:25.408088 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:35:25.408184 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 17:35:25.408316 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8388,"mutex_wait_us":25,"thread_start_us":9992,"threads_started":1}
I20260923 17:35:25.408738 28939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:25.390438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.20.240.125:43291)
0923 17:35:25.400597 (+ 10159us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:25.400603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:25.400622 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:35:25.400704 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:25.400710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:25.400782 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:25.400880 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:25.402099 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:25.404931 (+ 2832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:25.406840 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:25.407704 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:25.407852 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:25.407924 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:25.408214 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 17:35:25.408422 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":9671,"thread_start_us":9491,"threads_started":1}
I20260923 17:35:25.409021 28402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90f6ec2f0d9740be81aa4929afc03075" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "92f052fe126b402c805a99e40f9296c0" is_pre_election: true
I20260923 17:35:25.409238 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92f052fe126b402c805a99e40f9296c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "90f6ec2f0d9740be81aa4929afc03075" is_pre_election: true
I20260923 17:35:25.409374 28402 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90f6ec2f0d9740be81aa4929afc03075 in term 1.
I20260923 17:35:25.409467 28212 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92f052fe126b402c805a99e40f9296c0 in term 1.
I20260923 17:35:25.409826 28189 leader_election.cc:302] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90f6ec2f0d9740be81aa4929afc03075, 92f052fe126b402c805a99e40f9296c0; no voters: 9dc5d7a24a374aefa3e191059db69da4
I20260923 17:35:25.409881 28375 leader_election.cc:302] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90f6ec2f0d9740be81aa4929afc03075, 92f052fe126b402c805a99e40f9296c0; no voters: 9dc5d7a24a374aefa3e191059db69da4
I20260923 17:35:25.410140 28936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 17:35:25.410187 28935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 17:35:25.410259 28936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9dc5d7a24a374aefa3e191059db69da4)
I20260923 17:35:25.410295 28935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9dc5d7a24a374aefa3e191059db69da4)
I20260923 17:35:25.410342 28936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:35:25.410382 28935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:35:25.412401 28935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } }
I20260923 17:35:25.412408 28936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } }
I20260923 17:35:25.412992 28936 leader_election.cc:288] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [CANDIDATE]: Term 2 election: Requested vote from peers 9dc5d7a24a374aefa3e191059db69da4 (127.20.240.126:36193), 92f052fe126b402c805a99e40f9296c0 (127.20.240.124:33839)
I20260923 17:35:25.412982 28935 leader_election.cc:288] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [CANDIDATE]: Term 2 election: Requested vote from peers 9dc5d7a24a374aefa3e191059db69da4 (127.20.240.126:36193), 90f6ec2f0d9740be81aa4929afc03075 (127.20.240.125:43291)
I20260923 17:35:25.413455 28937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:25.413097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:48044)
0923 17:35:25.413220 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:25.413304 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 17:35:25.413666 28940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:25.413300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:48054)
0923 17:35:25.413436 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:25.413521 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 17:35:25.413774 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92f052fe126b402c805a99e40f9296c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "90f6ec2f0d9740be81aa4929afc03075"
I20260923 17:35:25.413854 28402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90f6ec2f0d9740be81aa4929afc03075" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "92f052fe126b402c805a99e40f9296c0"
I20260923 17:35:25.414016 28212 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 92f052fe126b402c805a99e40f9296c0 in current term 2: Already voted for candidate 90f6ec2f0d9740be81aa4929afc03075 in this term.
I20260923 17:35:25.414109 28402 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 90f6ec2f0d9740be81aa4929afc03075 in current term 2: Already voted for candidate 92f052fe126b402c805a99e40f9296c0 in this term.
I20260923 17:35:25.415011 28940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:25.414710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:48064)
0923 17:35:25.414840 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:25.414904 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 17:35:25.415091 28937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:25.414708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:48078)
0923 17:35:25.414868 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:25.414953 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 17:35:25.415199 28188 leader_election.cc:334] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9dc5d7a24a374aefa3e191059db69da4 (127.20.240.126:36193): Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
W20260923 17:35:25.415282 28377 leader_election.cc:334] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9dc5d7a24a374aefa3e191059db69da4 (127.20.240.126:36193): Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
I20260923 17:35:25.415307 28188 leader_election.cc:302] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90f6ec2f0d9740be81aa4929afc03075; no voters: 92f052fe126b402c805a99e40f9296c0, 9dc5d7a24a374aefa3e191059db69da4
I20260923 17:35:25.415398 28377 leader_election.cc:302] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92f052fe126b402c805a99e40f9296c0; no voters: 90f6ec2f0d9740be81aa4929afc03075, 9dc5d7a24a374aefa3e191059db69da4
I20260923 17:35:25.415540 28936 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 17:35:25.415689 28935 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260923 17:35:25.874125 28945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:25.873317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.67:51663)
0923 17:35:25.873789 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:25.873901 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":269,"threads_started":1}
I20260923 17:35:25.877177 28946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:25.876413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.65:59705)
0923 17:35:25.876907 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:25.876992 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":249,"threads_started":1}
I20260923 17:35:25.881400 28947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:25.880585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.66:41097)
0923 17:35:25.881140 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:25.881235 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":222,"threads_started":1}
I20260923 17:35:26.875891 28948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:26.874964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.67:59025)
0923 17:35:26.875455 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:26.875633 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":278,"threads_started":1}
I20260923 17:35:26.878816 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:26.878084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.65:47097)
0923 17:35:26.878543 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:26.878621 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":236,"threads_started":1}
I20260923 17:35:26.883040 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:26.882319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.66:47083)
0923 17:35:26.882737 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:26.882818 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":219,"threads_started":1}
I20260923 17:35:27.128773 28951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:35:27.129037 28951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } }
I20260923 17:35:27.129742 28951 leader_election.cc:288] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9dc5d7a24a374aefa3e191059db69da4 (127.20.240.126:36193), 90f6ec2f0d9740be81aa4929afc03075 (127.20.240.125:43291)
I20260923 17:35:27.130342 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92f052fe126b402c805a99e40f9296c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "90f6ec2f0d9740be81aa4929afc03075" is_pre_election: true
I20260923 17:35:27.130585 28212 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92f052fe126b402c805a99e40f9296c0 in term 2.
I20260923 17:35:27.130815 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.129927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:48322)
0923 17:35:27.130498 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.130625 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":332,"threads_started":1}
I20260923 17:35:27.131064 28375 leader_election.cc:302] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90f6ec2f0d9740be81aa4929afc03075, 92f052fe126b402c805a99e40f9296c0; no voters:
I20260923 17:35:27.131402 28951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260923 17:35:27.131562 28951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:35:27.131671 28951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 2 FOLLOWER]: Advancing to term 3
I20260923 17:35:27.132403 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.132010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:48332)
0923 17:35:27.132166 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.132258 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 17:35:27.132663 28377 leader_election.cc:334] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9dc5d7a24a374aefa3e191059db69da4 (127.20.240.126:36193): Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
I20260923 17:35:27.134110 28951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } }
I20260923 17:35:27.134656 28951 leader_election.cc:288] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [CANDIDATE]: Term 3 election: Requested vote from peers 9dc5d7a24a374aefa3e191059db69da4 (127.20.240.126:36193), 90f6ec2f0d9740be81aa4929afc03075 (127.20.240.125:43291)
I20260923 17:35:27.135298 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.134877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:48342)
0923 17:35:27.135013 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.135122 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 17:35:27.135295 28212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "92f052fe126b402c805a99e40f9296c0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "90f6ec2f0d9740be81aa4929afc03075"
I20260923 17:35:27.135545 28212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 2 FOLLOWER]: Advancing to term 3
I20260923 17:35:27.136162 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.135860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:48348)
0923 17:35:27.135988 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.136053 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 17:35:27.136307 28377 leader_election.cc:334] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9dc5d7a24a374aefa3e191059db69da4 (127.20.240.126:36193): Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
I20260923 17:35:27.137794 28212 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92f052fe126b402c805a99e40f9296c0 in term 3.
I20260923 17:35:27.138199 28375 leader_election.cc:302] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 90f6ec2f0d9740be81aa4929afc03075, 92f052fe126b402c805a99e40f9296c0; no voters: 9dc5d7a24a374aefa3e191059db69da4
I20260923 17:35:27.138541 28951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 17:35:27.139139 28951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [term 3 LEADER]: Becoming Leader. State: Replica: 92f052fe126b402c805a99e40f9296c0, State: Running, Role: LEADER
I20260923 17:35:27.139601 28951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } }
I20260923 17:35:27.141783 28954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92f052fe126b402c805a99e40f9296c0. Latest consensus state: current_term: 3 leader_uuid: "92f052fe126b402c805a99e40f9296c0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } } }
I20260923 17:35:27.142104 28954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:27.142726 28956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:35:27.146582 28956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:35:27.147805 28956 catalog_manager.cc:1259] Loaded cluster ID: f902ec2a20da4b6f917072be6129bc21
I20260923 17:35:27.147876 28956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:35:27.149003 28956 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:35:27.149804 28956 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: Loaded TSK: 0
I20260923 17:35:27.150627 28956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:35:27.236305 28390 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:35:27.244354 28390 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:35:27.248920 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.248432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:48382)
0923 17:35:27.248628 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.248727 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 17:35:27.249020 28212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [term 3 FOLLOWER]: Refusing update from remote peer 92f052fe126b402c805a99e40f9296c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260923 17:35:27.249797 28954 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260923 17:35:27.249995 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.249634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:48396)
0923 17:35:27.249803 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.249882 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 17:35:27.250211 28377 consensus_peers.cc:487] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 -> Peer 9dc5d7a24a374aefa3e191059db69da4 (127.20.240.126:36193): Couldn't send request to peer 9dc5d7a24a374aefa3e191059db69da4. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 17:35:27.253262 28958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92f052fe126b402c805a99e40f9296c0. Latest consensus state: current_term: 3 leader_uuid: "92f052fe126b402c805a99e40f9296c0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } } }
I20260923 17:35:27.253551 28958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:35:27.254508 28954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "92f052fe126b402c805a99e40f9296c0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } } }
I20260923 17:35:27.254827 28954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:27.259750 28958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "92f052fe126b402c805a99e40f9296c0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "9dc5d7a24a374aefa3e191059db69da4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36193 } } peers { permanent_uuid: "90f6ec2f0d9740be81aa4929afc03075" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 43291 } attrs { promote: false } } peers { permanent_uuid: "92f052fe126b402c805a99e40f9296c0" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 33839 } attrs { promote: false } } }
I20260923 17:35:27.260211 28958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f6ec2f0d9740be81aa4929afc03075 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:35:27.271232 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.265117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.20.240.67:45969)
0923 17:35:27.265612 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:27.265616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:27.265632 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:35:27.265693 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:27.265698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:27.265766 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:27.266681 (+ 915us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:27.266872 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.267843 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.269416 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.270221 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.270426 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:27.270541 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:27.270767 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 17:35:27.270902 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":262,"threads_started":1}
I20260923 17:35:27.271238 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.264980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:45969 (local address 127.0.0.1:39064)
0923 17:35:27.265128 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.265160 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:27.265286 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:27.265937 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:27.265942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:27.265973 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:27.266343 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:27.266358 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.267979 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.267986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:27.269262 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:27.269275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.270425 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.270432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:27.270615 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:27.270648 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:35:27.270730 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:35:27.270868 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 17:35:27.274402 28792 tablet_service.cc:1467] Processing CreateTablet for tablet 3b89206372334b178330b7d28d20e749 (DEFAULT_TABLE table=test-table [id=5a08a60ed9554f29a9eb7ba547a49b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:27.274962 28792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b89206372334b178330b7d28d20e749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:27.277977 28952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.265792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:37703 (local address 127.0.0.1:36508)
0923 17:35:27.271498 (+ 5706us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.271517 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:27.271624 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:27.272237 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:27.272242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:27.272269 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:27.272451 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:27.272464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.273942 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.273946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:27.275906 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:27.275920 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.277304 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.277308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:27.277438 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:27.277461 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:35:27.277540 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 17:35:27.277679 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5605,"thread_start_us":6041,"threads_started":1}
I20260923 17:35:27.278064 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.265935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36508 (local address 127.20.240.65:37703)
0923 17:35:27.266628 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:27.266634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:27.271939 (+ 5305us) server_negotiation.cc:407] Connection header received
0923 17:35:27.272028 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:27.272032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:27.272102 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:27.272202 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:27.272602 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.273798 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.276091 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.277016 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.277096 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:27.277185 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:27.277599 (+ 414us) server_negotiation.cc:299] Negotiation successful
0923 17:35:27.277750 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":341,"threads_started":1}
I20260923 17:35:27.278836 28505 tablet_service.cc:1467] Processing CreateTablet for tablet 3b89206372334b178330b7d28d20e749 (DEFAULT_TABLE table=test-table [id=5a08a60ed9554f29a9eb7ba547a49b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:27.279333 28505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b89206372334b178330b7d28d20e749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:27.282673 28968 tablet_bootstrap.cc:492] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03: Bootstrap starting.
I20260923 17:35:27.284502 28968 tablet_bootstrap.cc:654] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:27.285678 28969 tablet_bootstrap.cc:492] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba: Bootstrap starting.
I20260923 17:35:27.287400 28969 tablet_bootstrap.cc:654] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:27.287763 28968 tablet_bootstrap.cc:492] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03: No bootstrap required, opened a new log
I20260923 17:35:27.287918 28968 ts_tablet_manager.cc:1397] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:35:27.288784 28968 raft_consensus.cc:348] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } }
I20260923 17:35:27.289058 28968 raft_consensus.cc:374] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:27.289113 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.272355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.20.240.66:35803)
0923 17:35:27.282613 (+ 10258us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:27.282619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:27.282641 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:35:27.282719 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:27.282724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:27.282808 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:27.282926 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:27.283594 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.284647 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.287065 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.288169 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.288290 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:27.288396 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:27.288666 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 17:35:27.288801 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10120,"thread_start_us":10000,"threads_started":1}
I20260923 17:35:27.289162 28968 raft_consensus.cc:729] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494adc561a604d76954288bed8325b03, State: Initialized, Role: FOLLOWER
I20260923 17:35:27.289403 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.272184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:35803 (local address 127.0.0.1:35274)
0923 17:35:27.272687 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.272704 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:27.272799 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:27.282993 (+ 10194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:27.282999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:27.283031 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:27.283356 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:27.283371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.284797 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.284803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:27.286856 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:27.286871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.288346 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.288352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:27.288520 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:27.288544 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:35:27.288626 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:35:27.288768 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381}
I20260923 17:35:27.289474 28968 consensus_queue.cc:260] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } }
I20260923 17:35:27.290150 28650 tablet_service.cc:1467] Processing CreateTablet for tablet 3b89206372334b178330b7d28d20e749 (DEFAULT_TABLE table=test-table [id=5a08a60ed9554f29a9eb7ba547a49b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:27.290539 28650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b89206372334b178330b7d28d20e749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:27.293586 28968 ts_tablet_manager.cc:1428] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260923 17:35:27.295655 28948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.294220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.67:52773)
0923 17:35:27.295390 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.295492 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1018}
I20260923 17:35:27.297115 28857 heartbeater.cc:503] Master 127.20.240.124:33839 was elected leader, sending a full tablet report...
I20260923 17:35:27.297493 28973 tablet_bootstrap.cc:492] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314: Bootstrap starting.
I20260923 17:35:27.297838 28948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.297354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.67:53687)
0923 17:35:27.297592 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.297682 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260923 17:35:27.299849 28969 tablet_bootstrap.cc:492] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba: No bootstrap required, opened a new log
I20260923 17:35:27.300022 28969 ts_tablet_manager.cc:1397] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260923 17:35:27.300356 28973 tablet_bootstrap.cc:654] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:27.300935 28969 raft_consensus.cc:348] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } }
I20260923 17:35:27.301218 28969 raft_consensus.cc:374] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:27.301321 28969 raft_consensus.cc:729] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dbaafbf88034b6da6cfd705d4849bba, State: Initialized, Role: FOLLOWER
I20260923 17:35:27.301608 28969 consensus_queue.cc:260] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } }
I20260923 17:35:27.302423 28969 ts_tablet_manager.cc:1428] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260923 17:35:27.302459 28973 tablet_bootstrap.cc:492] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314: No bootstrap required, opened a new log
I20260923 17:35:27.302600 28973 ts_tablet_manager.cc:1397] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 17:35:27.303463 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.302986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.65:57471)
0923 17:35:27.303222 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.303314 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260923 17:35:27.303399 28973 raft_consensus.cc:348] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } }
I20260923 17:35:27.303623 28973 raft_consensus.cc:374] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:27.303706 28973 raft_consensus.cc:729] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 841cba3f424b4c6ea31bb73a14f17314, State: Initialized, Role: FOLLOWER
I20260923 17:35:27.303942 28973 consensus_queue.cc:260] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } }
I20260923 17:35:27.304796 28570 heartbeater.cc:503] Master 127.20.240.124:33839 was elected leader, sending a full tablet report...
I20260923 17:35:27.305763 28973 ts_tablet_manager.cc:1428] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 17:35:27.306463 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.306007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.66:41427)
0923 17:35:27.306206 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.306310 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 17:35:27.307879 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.307451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.65:32917)
0923 17:35:27.307620 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.307712 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 17:35:27.308076 28715 heartbeater.cc:503] Master 127.20.240.124:33839 was elected leader, sending a full tablet report...
I20260923 17:35:27.308445 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.306996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.66:48519)
0923 17:35:27.308258 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.308325 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1178}
I20260923 17:35:27.421833 28975 raft_consensus.cc:482] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:35:27.422055 28975 raft_consensus.cc:504] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } }
I20260923 17:35:27.422791 28975 leader_election.cc:288] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 494adc561a604d76954288bed8325b03 (127.20.240.67:45969), 841cba3f424b4c6ea31bb73a14f17314 (127.20.240.66:35803)
I20260923 17:35:27.423465 28812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b89206372334b178330b7d28d20e749" candidate_uuid: "4dbaafbf88034b6da6cfd705d4849bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494adc561a604d76954288bed8325b03" is_pre_election: true
I20260923 17:35:27.423632 28670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b89206372334b178330b7d28d20e749" candidate_uuid: "4dbaafbf88034b6da6cfd705d4849bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841cba3f424b4c6ea31bb73a14f17314" is_pre_election: true
I20260923 17:35:27.423759 28812 raft_consensus.cc:2454] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbaafbf88034b6da6cfd705d4849bba in term 0.
I20260923 17:35:27.423918 28670 raft_consensus.cc:2454] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dbaafbf88034b6da6cfd705d4849bba in term 0.
I20260923 17:35:27.424328 28458 leader_election.cc:302] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 494adc561a604d76954288bed8325b03, 4dbaafbf88034b6da6cfd705d4849bba; no voters:
I20260923 17:35:27.424598 28975 raft_consensus.cc:2781] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:35:27.424726 28975 raft_consensus.cc:482] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:35:27.424815 28975 raft_consensus.cc:3037] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:27.426905 28975 raft_consensus.cc:504] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } }
I20260923 17:35:27.427565 28975 leader_election.cc:288] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [CANDIDATE]: Term 1 election: Requested vote from peers 494adc561a604d76954288bed8325b03 (127.20.240.67:45969), 841cba3f424b4c6ea31bb73a14f17314 (127.20.240.66:35803)
I20260923 17:35:27.428211 28812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b89206372334b178330b7d28d20e749" candidate_uuid: "4dbaafbf88034b6da6cfd705d4849bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "494adc561a604d76954288bed8325b03"
I20260923 17:35:27.428233 28670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b89206372334b178330b7d28d20e749" candidate_uuid: "4dbaafbf88034b6da6cfd705d4849bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "841cba3f424b4c6ea31bb73a14f17314"
I20260923 17:35:27.428404 28812 raft_consensus.cc:3037] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:27.428472 28670 raft_consensus.cc:3037] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:27.430508 28812 raft_consensus.cc:2454] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbaafbf88034b6da6cfd705d4849bba in term 1.
I20260923 17:35:27.430552 28670 raft_consensus.cc:2454] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dbaafbf88034b6da6cfd705d4849bba in term 1.
I20260923 17:35:27.430994 28458 leader_election.cc:302] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dbaafbf88034b6da6cfd705d4849bba, 841cba3f424b4c6ea31bb73a14f17314; no voters:
I20260923 17:35:27.431342 28975 raft_consensus.cc:2781] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:27.431761 28975 raft_consensus.cc:686] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [term 1 LEADER]: Becoming Leader. State: Replica: 4dbaafbf88034b6da6cfd705d4849bba, State: Running, Role: LEADER
I20260923 17:35:27.432384 28975 consensus_queue.cc:237] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } }
I20260923 17:35:27.432790 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.432336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.65:38389)
0923 17:35:27.432527 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.432635 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 17:35:27.435566 28390 catalog_manager.cc:5495] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dbaafbf88034b6da6cfd705d4849bba (127.20.240.65). New cstate: current_term: 1 leader_uuid: "4dbaafbf88034b6da6cfd705d4849bba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dbaafbf88034b6da6cfd705d4849bba" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 } health_report { overall_health: UNKNOWN } } }
I20260923 17:35:27.467370 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.463006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.20.240.65:37703)
0923 17:35:27.463240 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:27.463244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:27.463262 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:35:27.463324 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:27.463328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:27.463393 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:27.463469 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:35:27.463893 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.464601 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.465841 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.466325 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.466420 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:27.466752 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:35:27.466846 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:35:27.466905 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:27.467025 (+ 120us) server_negotiation.cc:299] Negotiation successful
0923 17:35:27.467105 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:35:27.470337 28670 raft_consensus.cc:1261] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314 [term 1 FOLLOWER]: Refusing update from remote peer 4dbaafbf88034b6da6cfd705d4849bba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:27.470366 28812 raft_consensus.cc:1261] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03 [term 1 FOLLOWER]: Refusing update from remote peer 4dbaafbf88034b6da6cfd705d4849bba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:27.470954 28977 consensus_queue.cc:1035] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [LEADER]: Connected to new peer: Peer: permanent_uuid: "494adc561a604d76954288bed8325b03" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:27.471269 28975 consensus_queue.cc:1035] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [LEADER]: Connected to new peer: Peer: permanent_uuid: "841cba3f424b4c6ea31bb73a14f17314" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:27.473497 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.472992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.66:38231)
0923 17:35:27.473171 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.473272 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 17:35:27.474786 28948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.474040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.67:39675)
0923 17:35:27.474323 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.474410 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260923 17:35:27.478068 28950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.477534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.66:36901)
0923 17:35:27.477782 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.477876 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260923 17:35:27.480014 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.479534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.65:40713)
0923 17:35:27.479758 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.479847 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260923 17:35:27.483361 28949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.482838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.65:55305)
0923 17:35:27.483082 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.483191 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260923 17:35:27.484826 28390 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:27.485152 28390 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:27.487761 28948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.487192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.20.240.67:56593)
0923 17:35:27.487495 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.487587 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260923 17:35:27.493124 28390 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92f052fe126b402c805a99e40f9296c0: Sending DeleteTablet for 3 replicas of tablet 3b89206372334b178330b7d28d20e749
I20260923 17:35:27.494117 28792 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b89206372334b178330b7d28d20e749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:27 UTC) from {username='oryx', principal='oryx/127.20.240.124@KRBTEST.COM'} at 127.0.0.1:39064
I20260923 17:35:27.494164 28791 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b89206372334b178330b7d28d20e749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:27 UTC) from {username='oryx', principal='oryx/127.20.240.124@KRBTEST.COM'} at 127.0.0.1:39064
I20260923 17:35:27.494199 28650 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b89206372334b178330b7d28d20e749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:27 UTC) from {username='oryx', principal='oryx/127.20.240.124@KRBTEST.COM'} at 127.0.0.1:35274
I20260923 17:35:27.494779 28505 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b89206372334b178330b7d28d20e749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:27 UTC) from {username='oryx', principal='oryx/127.20.240.124@KRBTEST.COM'} at 127.0.0.1:36508
I20260923 17:35:27.495633 28990 tablet_replica.cc:330] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314: stopping tablet replica
I20260923 17:35:27.495801 28990 raft_consensus.cc:2229] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:27.496061 28990 raft_consensus.cc:2258] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:27.497398 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 28393
I20260923 17:35:27.497399 28990 ts_tablet_manager.cc:1905] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:27.502838 28990 ts_tablet_manager.cc:1918] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:27.502976 28990 log.cc:1196] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/3b89206372334b178330b7d28d20e749
I20260923 17:35:27.503387 28990 ts_tablet_manager.cc:1939] T 3b89206372334b178330b7d28d20e749 P 841cba3f424b4c6ea31bb73a14f17314: Deleting consensus metadata
I20260923 17:35:27.504400 28375 catalog_manager.cc:4841] TS 841cba3f424b4c6ea31bb73a14f17314 (127.20.240.66:35803): tablet 3b89206372334b178330b7d28d20e749 (table test-table [id=5a08a60ed9554f29a9eb7ba547a49b97]) successfully deleted
I20260923 17:35:27.506807 28991 tablet_replica.cc:330] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba: stopping tablet replica
I20260923 17:35:27.507023 28991 raft_consensus.cc:2229] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:35:27.507471 28991 raft_consensus.cc:2258] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:27.508810 28991 ts_tablet_manager.cc:1905] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:27.510913 28989 tablet_replica.cc:330] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03: stopping tablet replica
I20260923 17:35:27.511119 28989 raft_consensus.cc:2229] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:27.511466 28989 raft_consensus.cc:2258] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:27.512538 28989 ts_tablet_manager.cc:1905] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:27.515697 28991 ts_tablet_manager.cc:1918] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:27.515851 28991 log.cc:1196] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/3b89206372334b178330b7d28d20e749
I20260923 17:35:27.516350 28991 ts_tablet_manager.cc:1939] T 3b89206372334b178330b7d28d20e749 P 4dbaafbf88034b6da6cfd705d4849bba: Deleting consensus metadata
I20260923 17:35:27.517649 28375 catalog_manager.cc:4841] TS 4dbaafbf88034b6da6cfd705d4849bba (127.20.240.65:37703): tablet 3b89206372334b178330b7d28d20e749 (table test-table [id=5a08a60ed9554f29a9eb7ba547a49b97]) successfully deleted
I20260923 17:35:27.518428 28989 ts_tablet_manager.cc:1918] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:27.518538 28989 log.cc:1196] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/3b89206372334b178330b7d28d20e749
I20260923 17:35:27.518888 28989 ts_tablet_manager.cc:1939] T 3b89206372334b178330b7d28d20e749 P 494adc561a604d76954288bed8325b03: Deleting consensus metadata
I20260923 17:35:27.519915 28375 catalog_manager.cc:4841] TS 494adc561a604d76954288bed8325b03 (127.20.240.67:45969): tablet 3b89206372334b178330b7d28d20e749 (table test-table [id=5a08a60ed9554f29a9eb7ba547a49b97]) successfully deleted
W20260923 17:35:27.520582 28375 catalog_manager.cc:4808] TS 494adc561a604d76954288bed8325b03 (127.20.240.67:45969): delete failed for tablet 3b89206372334b178330b7d28d20e749 because the tablet was not found. No further retry: Not found: Tablet not found: 3b89206372334b178330b7d28d20e749
I20260923 17:35:27.520772 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.499555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.20.240.65:37703)
0923 17:35:27.499744 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:27.499750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:27.507725 (+ 7975us) server_negotiation.cc:407] Connection header received
0923 17:35:27.507976 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:27.507988 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:27.508116 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:27.508218 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:27.509759 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.510512 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.511957 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.512364 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.512469 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:27.514853 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:27.514866 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:27.514882 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:27.514913 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:27.517948 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:27.518561 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:27.518570 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:27.518572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:27.518709 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:27.519040 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:27.519046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:27.519049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:27.519545 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:27.519856 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:27.520186 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 17:35:27.520403 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:35:27.521386 28565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:27.639734 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 28393
I20260923 17:35:27.661501 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 28582
I20260923 17:35:27.675226 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.662316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.20.240.66:35803)
0923 17:35:27.662468 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:27.662473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:27.662530 (+ 57us) server_negotiation.cc:407] Connection header received
0923 17:35:27.662741 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:27.662746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:27.662856 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:27.662938 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:27.664433 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.665159 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.666437 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.666839 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.666951 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:27.669548 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:27.669564 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:27.669580 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:27.669614 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:27.672510 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:27.673169 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:27.673177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:27.673179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:27.673317 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:27.673685 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:27.673689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:27.673691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:27.674048 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:27.674341 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:27.674610 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 17:35:27.674830 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:35:27.675876 28710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:27.753086 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.752568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:48400)
0923 17:35:27.752761 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.752863 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 17:35:27.754410 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.754069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36193 (local address 127.0.0.1:48406)
0923 17:35:27.754206 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:27.754291 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 17:35:27.770012 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 28582
I20260923 17:35:27.784682 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 28728
I20260923 17:35:27.798686 28996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:27.785568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39068 (local address 127.20.240.67:45969)
0923 17:35:27.786082 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:27.786087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:27.786102 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:35:27.786156 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:27.786160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:27.786225 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:27.786318 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:27.787746 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.788471 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.789679 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:27.790093 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:27.790216 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:27.792744 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:27.792766 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:27.792782 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:27.792819 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:27.795867 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:27.796676 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:27.796684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:27.796686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:27.796834 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:27.797200 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:27.797204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:27.797206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:27.797551 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:27.797825 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:27.798129 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 17:35:27.798350 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":269,"threads_started":1}
I20260923 17:35:27.799340 28852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:27.894084 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 28728
I20260923 17:35:27.907946 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 28064
I20260923 17:35:27.909029 28232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:28.026057 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 28064
I20260923 17:35:28.042989 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 28166
I20260923 17:35:28.044130 28422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:28.181970 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:28.181484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:43291 (local address 127.0.0.1:53228)
0923 17:35:28.181647 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:28.181738 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:43291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 17:35:28.181982 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 28166
2026-09-23T17:35:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11419 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:35:28 dist-test-slave-ppjl krb5kdc[29002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:35:28 dist-test-slave-ppjl krb5kdc[29002](info): set up 2 sockets
Sep 23 17:35:28 dist-test-slave-ppjl krb5kdc[29002](info): commencing operation
krb5kdc: starting...
W20260923 17:35:29.200541 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:35:29 dist-test-slave-ppjl krb5kdc[29002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:35:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:35:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.126.keytab.
Entry for principal oryx/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.126.keytab.
I20260923 17:35:29.414718 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:43579
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.126.keytab
--principal=oryx/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:43579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:29.738087 29015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:29.738402 29015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:29.738524 29015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:29.746063 29015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:35:29.746187 29015 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 17:35:29.746232 29015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:29.746315 29015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:29.746387 29015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:35:29.746477 29015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:35:29.757433 29015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:43579
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.126.keytab
--principal=oryx/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:43579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:29.758843 29015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:29.760274 29015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:35:29.773346 29015 server_base.cc:1034] running on GCE node
W20260923 17:35:29.773471 29023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:29.773252 29021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:29.773269 29020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:29.774683 29015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:29.776252 29015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:29.777485 29015 hybrid_clock.cc:648] HybridClock initialized: now 1790184929777421 us; error 36 us; skew 500 ppm
I20260923 17:35:29.777808 29015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:29 dist-test-slave-ppjl krb5kdc[29002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184929, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:29.780901 29015 init.cc:377] Logged in from keytab as oryx/127.20.240.126@KRBTEST.COM (short username oryx)
I20260923 17:35:29.782375 29015 webserver.cc:460] Webserver started at http://127.20.240.126:36781/ using document root <none> and password file <none>
I20260923 17:35:29.783200 29015 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:29.783318 29015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:29.783622 29015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:29.786594 29015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "c18b6de8a8f54fb196e56cbc636ef6b5"
format_stamp: "Formatted at 2026-09-23 17:35:29 on dist-test-slave-ppjl"
server_key: "90c6be01f8d7f3f79f33d13b4a5c71c1"
server_key_iv: "d44360efb50521057952c6aeb006a236"
server_key_version: "encryptionkey@0"
I20260923 17:35:29.787421 29015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "c18b6de8a8f54fb196e56cbc636ef6b5"
format_stamp: "Formatted at 2026-09-23 17:35:29 on dist-test-slave-ppjl"
server_key: "90c6be01f8d7f3f79f33d13b4a5c71c1"
server_key_iv: "d44360efb50521057952c6aeb006a236"
server_key_version: "encryptionkey@0"
I20260923 17:35:29.792476 29015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:35:29.795980 29030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:29.797394 29015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:35:29.797554 29015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "c18b6de8a8f54fb196e56cbc636ef6b5"
format_stamp: "Formatted at 2026-09-23 17:35:29 on dist-test-slave-ppjl"
server_key: "90c6be01f8d7f3f79f33d13b4a5c71c1"
server_key_iv: "d44360efb50521057952c6aeb006a236"
server_key_version: "encryptionkey@0"
I20260923 17:35:29.797777 29015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:29.813820 29015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:29.817893 29015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:29.818157 29015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:29.838487 29015 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:43579
I20260923 17:35:29.838511 29091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:43579 every 8 connection(s)
I20260923 17:35:29.839936 29015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:35:29.843947 29092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:29.845111 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 29015
I20260923 17:35:29.845290 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:35:29.845937 21441 external_mini_cluster.cc:1428] Setting key baec942bd2fdd9ddb519fb1160765beb
I20260923 17:35:29.858372 29092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5: Bootstrap starting.
Sep 23 17:35:29 dist-test-slave-ppjl krb5kdc[29002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
I20260923 17:35:29.861615 29092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:29.863011 29092 log.cc:824] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:29.866669 29092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5: No bootstrap required, opened a new log
I20260923 17:35:29.871346 29095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:29.848053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.20.240.126:43579)
0923 17:35:29.848924 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:29.848939 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:29.849001 (+ 62us) server_negotiation.cc:407] Connection header received
0923 17:35:29.850771 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:29.850806 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:29.851385 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:29.851856 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:29.853133 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:29.855220 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:29.856473 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:29.857548 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:29.857722 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:29.862109 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:29.862122 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:29.862143 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:29.862199 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:29.866289 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:29.867273 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:29.867282 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:29.867292 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:29.867506 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:29.867910 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:29.867915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:29.867918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:29.868666 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:29.868954 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:29.869356 (+ 402us) server_negotiation.cc:299] Negotiation successful
0923 17:35:29.869840 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":350,"threads_started":1}
I20260923 17:35:29.876358 29092 raft_consensus.cc:348] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18b6de8a8f54fb196e56cbc636ef6b5" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43579 } }
I20260923 17:35:29.876684 29092 raft_consensus.cc:374] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:29.876760 29092 raft_consensus.cc:729] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18b6de8a8f54fb196e56cbc636ef6b5, State: Initialized, Role: FOLLOWER
I20260923 17:35:29.877269 29092 consensus_queue.cc:260] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18b6de8a8f54fb196e56cbc636ef6b5" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43579 } }
I20260923 17:35:29.877466 29092 raft_consensus.cc:388] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:35:29.877562 29092 raft_consensus.cc:482] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:35:29.877679 29092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:29.880067 29092 raft_consensus.cc:504] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18b6de8a8f54fb196e56cbc636ef6b5" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43579 } }
I20260923 17:35:29.880586 29092 leader_election.cc:302] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c18b6de8a8f54fb196e56cbc636ef6b5; no voters:
I20260923 17:35:29.881232 29092 leader_election.cc:288] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:35:29.881348 29097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:29.882059 29097 raft_consensus.cc:686] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [term 1 LEADER]: Becoming Leader. State: Replica: c18b6de8a8f54fb196e56cbc636ef6b5, State: Running, Role: LEADER
I20260923 17:35:29.882599 29097 consensus_queue.cc:237] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18b6de8a8f54fb196e56cbc636ef6b5" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43579 } }
I20260923 17:35:29.883141 29092 sys_catalog.cc:564] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:35:29.885327 29098 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c18b6de8a8f54fb196e56cbc636ef6b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18b6de8a8f54fb196e56cbc636ef6b5" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43579 } } }
I20260923 17:35:29.885735 29098 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:29.885301 29099 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c18b6de8a8f54fb196e56cbc636ef6b5. Latest consensus state: current_term: 1 leader_uuid: "c18b6de8a8f54fb196e56cbc636ef6b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c18b6de8a8f54fb196e56cbc636ef6b5" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43579 } } }
I20260923 17:35:29.886055 29099 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:29.888005 29105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:35:29.893108 29105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:35:29.901903 29105 catalog_manager.cc:1347] Generated new cluster ID: b1a16a8456454072b799bf5595313782
I20260923 17:35:29.902042 29105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:35:29.929437 29105 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:35:29.930847 29105 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:35:29.946614 29105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5: Generated new TSK 0
I20260923 17:35:29.947525 29105 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.65.keytab.
Entry for principal oryx/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.65.keytab.
I20260923 17:35:30.006234 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43579
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.65.keytab
--principal=oryx/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:30.334272 29118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:30.334592 29118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:30.334733 29118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:30.342150 29118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:30.342275 29118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:30.342357 29118 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:35:30.342444 29118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:35:30.352341 29118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.65.keytab
--principal=oryx/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.240.126:43579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:30.353595 29118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:30.354887 29118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:30.367959 29126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:30.367982 29124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:30.368243 29123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:30.368736 29118 server_base.cc:1034] running on GCE node
I20260923 17:35:30.369329 29118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:30.370457 29118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:30.371703 29118 hybrid_clock.cc:648] HybridClock initialized: now 1790184930371629 us; error 39 us; skew 500 ppm
I20260923 17:35:30.372032 29118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:30 dist-test-slave-ppjl krb5kdc[29002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184930, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:30.375202 29118 init.cc:377] Logged in from keytab as oryx/127.20.240.65@KRBTEST.COM (short username oryx)
I20260923 17:35:30.376665 29118 webserver.cc:460] Webserver started at http://127.20.240.65:36289/ using document root <none> and password file <none>
I20260923 17:35:30.377358 29118 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:30.377450 29118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:30.377751 29118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:30.380704 29118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "4e4fb9691c284161ad175ef91014eea5"
format_stamp: "Formatted at 2026-09-23 17:35:30 on dist-test-slave-ppjl"
server_key: "579065bc5d1149bd755410a2bd38b9fe"
server_key_iv: "dbc98976d6bd3a8122c1282bcb547c09"
server_key_version: "encryptionkey@0"
I20260923 17:35:30.381448 29118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "4e4fb9691c284161ad175ef91014eea5"
format_stamp: "Formatted at 2026-09-23 17:35:30 on dist-test-slave-ppjl"
server_key: "579065bc5d1149bd755410a2bd38b9fe"
server_key_iv: "dbc98976d6bd3a8122c1282bcb547c09"
server_key_version: "encryptionkey@0"
I20260923 17:35:30.386699 29118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 17:35:30.390270 29133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:30.391831 29118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:35:30.392059 29118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "4e4fb9691c284161ad175ef91014eea5"
format_stamp: "Formatted at 2026-09-23 17:35:30 on dist-test-slave-ppjl"
server_key: "579065bc5d1149bd755410a2bd38b9fe"
server_key_iv: "dbc98976d6bd3a8122c1282bcb547c09"
server_key_version: "encryptionkey@0"
I20260923 17:35:30.392269 29118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:30.409497 29118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:30.413820 29118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:30.414085 29118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:30.416889 29118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:30.416978 29118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:30.417110 29118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:30.417172 29118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:35:30 dist-test-slave-ppjl krb5kdc[29002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184930, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
I20260923 17:35:30.440414 29142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:30.417454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:35265 (local address 127.20.240.126:43579)
0923 17:35:30.418080 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:30.418086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:30.419173 (+ 1087us) server_negotiation.cc:407] Connection header received
0923 17:35:30.421271 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:30.421278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:30.421369 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:30.421478 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:30.424221 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.425215 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.428235 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.428783 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.428938 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:30.433424 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:30.433430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:30.433433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:30.433469 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:30.436849 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:30.437613 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:30.437619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:30.437623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:30.437783 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:30.438269 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:30.438274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:30.438278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:30.438672 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:30.438823 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:30.439742 (+ 919us) server_negotiation.cc:299] Negotiation successful
0923 17:35:30.440005 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":369,"threads_started":1}
I20260923 17:35:30.441359 29143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:30.418097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:43579 (local address 127.20.240.65:35265)
0923 17:35:30.418964 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:30.419010 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:30.420971 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:30.421849 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:30.421866 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:30.422800 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:30.423937 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:30.423971 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.425411 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.425420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:30.428029 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:30.428052 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.429052 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.429065 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:30.429445 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:30.430276 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:30.430301 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:30.433161 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:30.437012 (+ 3851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:30.437019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:30.437033 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:30.437490 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:30.437943 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:30.437946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:30.437949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:30.438162 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:30.438818 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:30.438825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:30.439307 (+ 482us) client_negotiation.cc:769] Sending connection context
0923 17:35:30.439618 (+ 311us) client_negotiation.cc:240] Negotiation successful
0923 17:35:30.440120 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":353,"threads_started":1}
I20260923 17:35:30.462586 29207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:30.447441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:43579 (local address 127.20.240.65:51169)
0923 17:35:30.448132 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:30.448160 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:30.448319 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:30.448664 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:30.448670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:30.449288 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:30.449704 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:30.449717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.450700 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.450705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:30.452470 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:30.452488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.453515 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.453523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:30.453754 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:30.454592 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:30.454619 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:30.455898 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:30.459886 (+ 3988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:30.459891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:30.459895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:30.460504 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:30.460977 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:30.460982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:30.460984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:30.461102 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:30.461690 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:30.461694 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:30.461855 (+ 161us) client_negotiation.cc:769] Sending connection context
0923 17:35:30.461962 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 17:35:30.462209 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":357,"threads_started":1}
I20260923 17:35:30.462725 29142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:30.447367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:51169 (local address 127.20.240.126:43579)
0923 17:35:30.447559 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:30.447565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:30.448258 (+ 693us) server_negotiation.cc:407] Connection header received
0923 17:35:30.448450 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:30.448454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:30.448520 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:30.448601 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:30.449862 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.450591 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.452651 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.453280 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.453427 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:30.456115 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:30.456123 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:30.456128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:30.456173 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:30.459740 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:30.460678 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:30.460684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:30.460688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:30.460861 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:30.461199 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:30.461204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:30.461208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:30.461579 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:30.461738 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:30.462100 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 17:35:30.462361 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:35:30.470999 29118 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:34607
I20260923 17:35:30.471079 29253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:34607 every 8 connection(s)
I20260923 17:35:30.472321 29118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:35:30.474517 29254 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43579
I20260923 17:35:30.474766 29254 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:30.475526 29254 heartbeater.cc:511] Master 127.20.240.126:43579 requested a full tablet report, sending...
I20260923 17:35:30.477497 29047 ts_manager.cc:194] Registered new tserver with Master: 4e4fb9691c284161ad175ef91014eea5 (127.20.240.65:34607)
I20260923 17:35:30.477558 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 29118
I20260923 17:35:30.477738 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:35:30.478406 21441 external_mini_cluster.cc:1428] Setting key 7dba4f96773b63975f7e3a88971293d4
I20260923 17:35:30.479080 29047 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:35265
WARNING: no policy specified for oryx/127.20.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.66.keytab.
Entry for principal oryx/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.66.keytab.
I20260923 17:35:30.522898 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43579
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.66.keytab
--principal=oryx/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:30.844683 29258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:30.844964 29258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:30.845088 29258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:30.852636 29258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:30.852746 29258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:30.852854 29258 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:35:30.852948 29258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:35:30.862887 29258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.66.keytab
--principal=oryx/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.240.126:43579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:30.864280 29258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:30.865602 29258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:30.878803 29264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:30.878803 29266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:30.879011 29263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:30.879784 29258 server_base.cc:1034] running on GCE node
I20260923 17:35:30.880380 29258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:30.881400 29258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:30.882643 29258 hybrid_clock.cc:648] HybridClock initialized: now 1790184930882577 us; error 37 us; skew 500 ppm
I20260923 17:35:30.882938 29258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:30 dist-test-slave-ppjl krb5kdc[29002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184930, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:30.886093 29258 init.cc:377] Logged in from keytab as oryx/127.20.240.66@KRBTEST.COM (short username oryx)
I20260923 17:35:30.887681 29258 webserver.cc:460] Webserver started at http://127.20.240.66:37049/ using document root <none> and password file <none>
I20260923 17:35:30.888371 29258 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:30.888466 29258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:30.888769 29258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:30.891724 29258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "939711233b934a7692a3bd0151f4a8ec"
format_stamp: "Formatted at 2026-09-23 17:35:30 on dist-test-slave-ppjl"
server_key: "5e1ec6f306024baccbd09f728ba533c9"
server_key_iv: "0470e1c9066e004f68d11dc3f21316fd"
server_key_version: "encryptionkey@0"
I20260923 17:35:30.892510 29258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "939711233b934a7692a3bd0151f4a8ec"
format_stamp: "Formatted at 2026-09-23 17:35:30 on dist-test-slave-ppjl"
server_key: "5e1ec6f306024baccbd09f728ba533c9"
server_key_iv: "0470e1c9066e004f68d11dc3f21316fd"
server_key_version: "encryptionkey@0"
I20260923 17:35:30.897807 29258 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:35:30.901561 29273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:30.903034 29258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:35:30.903364 29258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "939711233b934a7692a3bd0151f4a8ec"
format_stamp: "Formatted at 2026-09-23 17:35:30 on dist-test-slave-ppjl"
server_key: "5e1ec6f306024baccbd09f728ba533c9"
server_key_iv: "0470e1c9066e004f68d11dc3f21316fd"
server_key_version: "encryptionkey@0"
I20260923 17:35:30.903590 29258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:30.920581 29258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:30.924930 29258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:30.925194 29258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:30.927920 29258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:30.928017 29258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:30.928145 29258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:30.928201 29258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:35:30 dist-test-slave-ppjl krb5kdc[29002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184930, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
I20260923 17:35:30.952281 29142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:30.928589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:41331 (local address 127.20.240.126:43579)
0923 17:35:30.928782 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:30.928787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:30.930269 (+ 1482us) server_negotiation.cc:407] Connection header received
0923 17:35:30.932938 (+ 2669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:30.932944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:30.933030 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:30.933120 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:30.935368 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.936369 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.938861 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.939498 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.939642 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:30.945049 (+ 5407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:30.945054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:30.945058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:30.945094 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:30.948069 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:30.949138 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:30.949145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:30.949148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:30.949326 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:30.949933 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:30.949939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:30.949942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:30.950335 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:30.950496 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:30.951662 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0923 17:35:30.951920 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:35:30.953475 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:30.929158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:43579 (local address 127.20.240.66:41331)
0923 17:35:30.930080 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:30.930130 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:30.932522 (+ 2392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:30.933445 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:30.933461 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:30.934243 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:30.935138 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:30.935197 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.936620 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.936626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:30.938689 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:30.938713 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.939782 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.939791 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:30.940187 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:30.941137 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:30.941178 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:30.944785 (+ 3607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:30.948241 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:30.948249 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:30.948269 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:30.948962 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:30.949473 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:30.949477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:30.949481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:30.949771 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:30.950506 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:30.950515 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:30.951140 (+ 625us) client_negotiation.cc:769] Sending connection context
0923 17:35:30.951523 (+ 383us) client_negotiation.cc:240] Negotiation successful
0923 17:35:30.952208 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":371,"threads_started":1}
I20260923 17:35:30.967592 29258 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:45963
I20260923 17:35:30.967617 29390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:45963 every 8 connection(s)
I20260923 17:35:30.968863 29258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:35:30.971067 29391 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43579
I20260923 17:35:30.971310 29391 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:30.972010 29391 heartbeater.cc:511] Master 127.20.240.126:43579 requested a full tablet report, sending...
I20260923 17:35:30.973320 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 29258
I20260923 17:35:30.973500 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:35:30.973601 29046 ts_manager.cc:194] Registered new tserver with Master: 939711233b934a7692a3bd0151f4a8ec (127.20.240.66:45963)
I20260923 17:35:30.974028 21441 external_mini_cluster.cc:1428] Setting key 7434ecd92c286186e1fab558a18f19e3
I20260923 17:35:30.974550 29046 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:41331
I20260923 17:35:30.985148 29142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:30.972463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:38599 (local address 127.20.240.126:43579)
0923 17:35:30.972622 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:30.972626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:30.973319 (+ 693us) server_negotiation.cc:407] Connection header received
0923 17:35:30.973606 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:30.973611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:30.973691 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:30.973794 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:30.974823 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.975830 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.977614 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.978000 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.978094 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:30.979987 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:30.979991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:30.979995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:30.980026 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:30.982431 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:30.983095 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:30.983100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:30.983104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:30.983282 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:30.983632 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:30.983637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:30.983640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:30.984004 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:30.984117 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:30.984560 (+ 443us) server_negotiation.cc:299] Negotiation successful
0923 17:35:30.984776 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:35:30.985194 29394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:30.972575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:43579 (local address 127.20.240.66:38599)
0923 17:35:30.973217 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:30.973238 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:30.973380 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:30.973831 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:30.973836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:30.974364 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:30.974687 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:30.974702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.975948 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.975954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:30.977468 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:30.977484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:30.978169 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:30.978175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:30.978325 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:30.978904 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:30.978924 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:30.979801 (+ 877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:30.982560 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:30.982565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:30.982570 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:30.982972 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:30.983381 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:30.983384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:30.983386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:30.983537 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:30.984107 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:30.984111 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:30.984309 (+ 198us) client_negotiation.cc:769] Sending connection context
0923 17:35:30.984433 (+ 124us) client_negotiation.cc:240] Negotiation successful
0923 17:35:30.984776 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":319,"threads_started":1}
WARNING: no policy specified for oryx/127.20.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.67.keytab.
Entry for principal oryx/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.67.keytab.
I20260923 17:35:31.021878 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43579
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.67.keytab
--principal=oryx/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:38565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:31.338539 29397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:31.338840 29397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:31.338935 29397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:31.346232 29397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:31.346340 29397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:31.346431 29397 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:35:31.346514 29397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:35:31.356400 29397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:38565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.67.keytab
--principal=oryx/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.240.126:43579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:31.357678 29397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:31.358966 29397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:31.372057 29405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:31.372042 29403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:31.372488 29402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:31.372933 29397 server_base.cc:1034] running on GCE node
I20260923 17:35:31.373675 29397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:31.374712 29397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:31.375948 29397 hybrid_clock.cc:648] HybridClock initialized: now 1790184931375870 us; error 45 us; skew 500 ppm
I20260923 17:35:31.376241 29397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:31 dist-test-slave-ppjl krb5kdc[29002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184931, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:31.379441 29397 init.cc:377] Logged in from keytab as oryx/127.20.240.67@KRBTEST.COM (short username oryx)
I20260923 17:35:31.380848 29397 webserver.cc:460] Webserver started at http://127.20.240.67:38127/ using document root <none> and password file <none>
I20260923 17:35:31.381538 29397 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:31.381635 29397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:31.381938 29397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:31.384989 29397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "b1b24ef4df9248569569bc764209741c"
format_stamp: "Formatted at 2026-09-23 17:35:31 on dist-test-slave-ppjl"
server_key: "bdb220baad94776af65dd493a3d5fe23"
server_key_iv: "733f17fcac4556556671e2105c27457f"
server_key_version: "encryptionkey@0"
I20260923 17:35:31.385731 29397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "b1b24ef4df9248569569bc764209741c"
format_stamp: "Formatted at 2026-09-23 17:35:31 on dist-test-slave-ppjl"
server_key: "bdb220baad94776af65dd493a3d5fe23"
server_key_iv: "733f17fcac4556556671e2105c27457f"
server_key_version: "encryptionkey@0"
I20260923 17:35:31.391238 29397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:35:31.394887 29412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:31.396610 29397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:35:31.396790 29397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "b1b24ef4df9248569569bc764209741c"
format_stamp: "Formatted at 2026-09-23 17:35:31 on dist-test-slave-ppjl"
server_key: "bdb220baad94776af65dd493a3d5fe23"
server_key_iv: "733f17fcac4556556671e2105c27457f"
server_key_version: "encryptionkey@0"
I20260923 17:35:31.397050 29397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:31.414801 29397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:31.419605 29397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:31.419862 29397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:31.422619 29397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:31.422725 29397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:31.422859 29397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:31.422937 29397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:35:31 dist-test-slave-ppjl krb5kdc[29002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184931, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
I20260923 17:35:31.445786 29142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:31.423470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:36451 (local address 127.20.240.126:43579)
0923 17:35:31.423661 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:31.423666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:31.425097 (+ 1431us) server_negotiation.cc:407] Connection header received
0923 17:35:31.427335 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:31.427342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:31.427429 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:31.427504 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:31.429934 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.430952 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.433182 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.433709 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.433818 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.438339 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:31.438344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:31.438347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:31.438402 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:31.441604 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:31.442534 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:31.442540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:31.442544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:31.442717 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:31.443428 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:31.443433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:31.443437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:31.443816 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:31.443958 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:31.445062 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0923 17:35:31.445357 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:35:31.446542 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:31.424090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:43579 (local address 127.20.240.67:36451)
0923 17:35:31.424894 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:31.424930 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:31.426947 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:31.428068 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:31.428080 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:31.428868 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:31.429752 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:31.429783 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.431116 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.431121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:31.432994 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:31.433011 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.433964 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.433972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:31.434213 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.434981 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:31.435019 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:31.438120 (+ 3101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:31.441739 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:31.441752 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:31.441770 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:31.442411 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:31.442882 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:31.442889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:31.442894 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:31.443271 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:31.443951 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:31.443960 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:31.444504 (+ 544us) client_negotiation.cc:769] Sending connection context
0923 17:35:31.444860 (+ 356us) client_negotiation.cc:240] Negotiation successful
0923 17:35:31.445342 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":336,"threads_started":1}
I20260923 17:35:31.465034 29397 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:46455
I20260923 17:35:31.465090 29531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:46455 every 8 connection(s)
I20260923 17:35:31.466653 29397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:35:31.468654 29532 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43579
I20260923 17:35:31.468840 29142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:31.453353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:39239 (local address 127.20.240.126:43579)
0923 17:35:31.453565 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:31.453572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:31.454069 (+ 497us) server_negotiation.cc:407] Connection header received
0923 17:35:31.454262 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:31.454267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:31.454346 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:31.454444 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:31.456074 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.456992 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.458496 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.459035 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.459152 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.462260 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:31.462267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:31.462272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:31.462315 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:31.465903 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:31.466556 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:31.466562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:31.466565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:31.466737 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:31.467075 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:31.467080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:31.467083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:31.467531 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:31.467697 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:31.468279 (+ 582us) server_negotiation.cc:299] Negotiation successful
0923 17:35:31.468506 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:35:31.468912 29532 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:31.469018 29498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:31.453315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:43579 (local address 127.20.240.67:39239)
0923 17:35:31.453950 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:31.453973 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:31.454142 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:31.454640 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:31.454644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:31.455367 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:31.455875 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:31.455894 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.457126 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.457132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:31.458367 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:31.458382 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.459342 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.459349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:31.459653 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.460575 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:31.460602 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:31.461968 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:31.466045 (+ 4077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:31.466050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:31.466054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:31.466437 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:31.466848 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:31.466851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:31.466854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:31.466971 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:31.467763 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:31.467767 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:31.467999 (+ 232us) client_negotiation.cc:769] Sending connection context
0923 17:35:31.468146 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 17:35:31.468520 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":309,"threads_started":1}
I20260923 17:35:31.469695 29532 heartbeater.cc:511] Master 127.20.240.126:43579 requested a full tablet report, sending...
I20260923 17:35:31.471184 29046 ts_manager.cc:194] Registered new tserver with Master: b1b24ef4df9248569569bc764209741c (127.20.240.67:46455)
I20260923 17:35:31.472326 29046 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:36451
I20260923 17:35:31.472960 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 29397
I20260923 17:35:31.473124 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:35:31.473537 21441 external_mini_cluster.cc:1428] Setting key 97980a9087be5d40dc77feb989ffd409
I20260923 17:35:31.476265 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:35:31.482874 29254 heartbeater.cc:503] Master 127.20.240.126:43579 was elected leader, sending a full tablet report...
Sep 23 17:35:31 dist-test-slave-ppjl krb5kdc[29002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:35:31 dist-test-slave-ppjl krb5kdc[29002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:31 dist-test-slave-ppjl krb5kdc[29002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:35:31.523444 29142 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:31.513614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.20.240.126:43579)
0923 17:35:31.513864 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:31.513871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:31.514027 (+ 156us) server_negotiation.cc:407] Connection header received
0923 17:35:31.514204 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:31.514208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:31.514278 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:31.514366 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:31.515803 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.516577 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.517721 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.518109 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.518203 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.523214 (+ 5011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32840: BlockingRecv error: recv got EOF from 127.0.0.1:32840 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 23 17:35:31 dist-test-slave-ppjl krb5kdc[29002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
I20260923 17:35:31.539741 29142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:31.526602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.20.240.126:43579)
0923 17:35:31.526763 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:31.526767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:31.527122 (+ 355us) server_negotiation.cc:407] Connection header received
0923 17:35:31.527344 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:31.527348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:31.527427 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:31.527510 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:31.528701 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.529466 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.530874 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.531412 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.531515 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.535106 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:31.535112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:31.535115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:31.535150 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:31.537525 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:31.538147 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:31.538151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:31.538154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:31.538285 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:31.538576 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:31.538579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:31.538581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:31.538834 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:31.539067 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:31.539313 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 17:35:31.539499 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:35:31.543963 29046 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:35:31.550710 29046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:35:31.590184 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:31.564390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48184 (local address 127.20.240.66:45963)
0923 17:35:31.580178 (+ 15788us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:31.580188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:31.582515 (+ 2327us) server_negotiation.cc:407] Connection header received
0923 17:35:31.582665 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:31.582674 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:31.582883 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:31.583072 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:31.583711 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.584897 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.587136 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.588896 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.589041 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.589268 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:31.589789 (+ 521us) server_negotiation.cc:299] Negotiation successful
0923 17:35:31.589944 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15624,"thread_start_us":15508,"threads_started":1}
I20260923 17:35:31.590183 29551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:31.563619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42754 (local address 127.20.240.67:46455)
0923 17:35:31.579865 (+ 16246us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:31.579876 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:31.581311 (+ 1435us) server_negotiation.cc:407] Connection header received
0923 17:35:31.581632 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:31.581641 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:31.581924 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:31.582122 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:31.583132 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.584949 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.587637 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.588986 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.589099 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.589295 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:31.589640 (+ 345us) server_negotiation.cc:299] Negotiation successful
0923 17:35:31.589815 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16055,"thread_start_us":15859,"threads_started":1}
I20260923 17:35:31.590543 29553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:31.563337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:46455 (local address 127.0.0.1:42754)
0923 17:35:31.581141 (+ 17804us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:31.581177 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:31.581395 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:31.582410 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:31.582432 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:31.582466 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:31.582880 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:31.582894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.585102 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.585107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:31.587470 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:31.587485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.589170 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.589176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:31.589376 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.589434 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 17:35:31.589571 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 17:35:31.589827 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17517,"mutex_wait_us":3,"thread_start_us":18791,"threads_started":1}
I20260923 17:35:31.590912 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:31.563940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:45963 (local address 127.0.0.1:48184)
0923 17:35:31.582390 (+ 18450us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:31.582405 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:31.582525 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:31.583122 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:31.583130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:31.583179 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:31.583562 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:31.583571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.585071 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.585075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:31.586969 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:31.586984 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.589411 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.589423 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:31.589538 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.589570 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:35:31.589669 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 17:35:31.590525 (+ 856us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18339,"mutex_wait_us":753,"thread_start_us":16814,"threads_started":1}
I20260923 17:35:31.590967 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:31.564146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37384 (local address 127.20.240.65:34607)
0923 17:35:31.580590 (+ 16444us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:31.580598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:31.582238 (+ 1640us) server_negotiation.cc:407] Connection header received
0923 17:35:31.582343 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:31.582350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:31.582588 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:31.582772 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:31.583477 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.585327 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.587827 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.589178 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.589312 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.589524 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:31.590588 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0923 17:35:31.590758 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16303,"thread_start_us":16172,"threads_started":1}
I20260923 17:35:31.591250 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:31.563816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34607 (local address 127.0.0.1:37384)
0923 17:35:31.581963 (+ 18147us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:31.581981 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:31.582107 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:31.582784 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:31.582793 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:31.582816 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:31.583286 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:31.583303 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.585503 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.585508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:31.587689 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:31.587699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.589664 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.589669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:31.589827 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.589854 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:35:31.589923 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 17:35:31.590041 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18022,"mutex_wait_us":1,"thread_start_us":17867,"threads_started":1}
I20260923 17:35:31.594508 29462 tablet_service.cc:1467] Processing CreateTablet for tablet abe671690e31487489854bb71c8657bb (DEFAULT_TABLE table=test-table [id=3ebec32de3324ec3bb287c9966b1bc72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:31.594511 29323 tablet_service.cc:1467] Processing CreateTablet for tablet abe671690e31487489854bb71c8657bb (DEFAULT_TABLE table=test-table [id=3ebec32de3324ec3bb287c9966b1bc72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:31.595702 29323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abe671690e31487489854bb71c8657bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:31.595826 29462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abe671690e31487489854bb71c8657bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:31.597891 29184 tablet_service.cc:1467] Processing CreateTablet for tablet abe671690e31487489854bb71c8657bb (DEFAULT_TABLE table=test-table [id=3ebec32de3324ec3bb287c9966b1bc72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:31.599288 29184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abe671690e31487489854bb71c8657bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:31.607761 29556 tablet_bootstrap.cc:492] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec: Bootstrap starting.
I20260923 17:35:31.610420 29557 tablet_bootstrap.cc:492] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c: Bootstrap starting.
I20260923 17:35:31.611920 29556 tablet_bootstrap.cc:654] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:31.613312 29556 log.cc:824] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:31.613701 29558 tablet_bootstrap.cc:492] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5: Bootstrap starting.
I20260923 17:35:31.614404 29557 tablet_bootstrap.cc:654] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:31.615757 29557 log.cc:824] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:31.616866 29558 tablet_bootstrap.cc:654] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:31.618070 29558 log.cc:824] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:31.618467 29557 tablet_bootstrap.cc:492] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c: No bootstrap required, opened a new log
I20260923 17:35:31.618736 29557 ts_tablet_manager.cc:1397] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 17:35:31.618925 29556 tablet_bootstrap.cc:492] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec: No bootstrap required, opened a new log
I20260923 17:35:31.619201 29556 ts_tablet_manager.cc:1397] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20260923 17:35:31.620954 29558 tablet_bootstrap.cc:492] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5: No bootstrap required, opened a new log
I20260923 17:35:31.621209 29558 ts_tablet_manager.cc:1397] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260923 17:35:31.626958 29556 raft_consensus.cc:348] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } }
I20260923 17:35:31.627383 29556 raft_consensus.cc:374] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:31.627493 29556 raft_consensus.cc:729] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939711233b934a7692a3bd0151f4a8ec, State: Initialized, Role: FOLLOWER
I20260923 17:35:31.628115 29556 consensus_queue.cc:260] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } }
I20260923 17:35:31.629161 29391 heartbeater.cc:503] Master 127.20.240.126:43579 was elected leader, sending a full tablet report...
I20260923 17:35:31.629333 29557 raft_consensus.cc:348] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } }
I20260923 17:35:31.629685 29556 ts_tablet_manager.cc:1428] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260923 17:35:31.629770 29557 raft_consensus.cc:374] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:31.629868 29557 raft_consensus.cc:729] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b24ef4df9248569569bc764209741c, State: Initialized, Role: FOLLOWER
I20260923 17:35:31.630439 29557 consensus_queue.cc:260] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } }
I20260923 17:35:31.631862 29532 heartbeater.cc:503] Master 127.20.240.126:43579 was elected leader, sending a full tablet report...
I20260923 17:35:31.632182 29557 ts_tablet_manager.cc:1428] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260923 17:35:31.634904 29558 raft_consensus.cc:348] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } }
I20260923 17:35:31.635425 29558 raft_consensus.cc:374] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:31.635531 29558 raft_consensus.cc:729] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4fb9691c284161ad175ef91014eea5, State: Initialized, Role: FOLLOWER
I20260923 17:35:31.636137 29558 consensus_queue.cc:260] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } }
I20260923 17:35:31.637542 29558 ts_tablet_manager.cc:1428] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
W20260923 17:35:31.718398 29533 tablet.cc:2369] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:35:31.721031 29392 tablet.cc:2369] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:35:31.725603 29255 tablet.cc:2369] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:35:31.964323 29564 raft_consensus.cc:482] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:35:31.964648 29564 raft_consensus.cc:504] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } }
I20260923 17:35:31.966497 29564 leader_election.cc:288] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1b24ef4df9248569569bc764209741c (127.20.240.67:46455), 939711233b934a7692a3bd0151f4a8ec (127.20.240.66:45963)
I20260923 17:35:31.975262 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:31.967217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:51279 (local address 127.20.240.66:45963)
0923 17:35:31.967949 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:31.967956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:31.969163 (+ 1207us) server_negotiation.cc:407] Connection header received
0923 17:35:31.969625 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:31.969630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:31.969711 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:31.969771 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:31.970532 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.971450 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.973502 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.974316 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.974493 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.974569 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:31.974837 (+ 268us) server_negotiation.cc:299] Negotiation successful
0923 17:35:31.974967 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545}
I20260923 17:35:31.975497 29551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:31.968571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:52977 (local address 127.20.240.67:46455)
0923 17:35:31.968740 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:31.968747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:31.968767 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:35:31.968943 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:31.968949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:31.969033 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:31.969096 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:31.969830 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.970561 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.973112 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.974363 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.974491 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.974572 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:31.975099 (+ 527us) server_negotiation.cc:299] Negotiation successful
0923 17:35:31.975261 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 17:35:31.975579 29565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:31.967022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:45963 (local address 127.20.240.65:51279)
0923 17:35:31.969028 (+ 2006us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:31.969047 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:31.969208 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:31.969953 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:31.969956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:31.969979 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:31.970345 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:31.970355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.971631 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.971637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:31.973351 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:31.973366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.974520 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.974526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:31.974686 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.974714 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 17:35:31.974799 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 17:35:31.974945 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1876,"mutex_wait_us":6,"thread_start_us":1119,"threads_started":1}
I20260923 17:35:31.976267 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:31.966981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:46455 (local address 127.20.240.65:52977)
0923 17:35:31.968537 (+ 1556us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:31.968560 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:31.968740 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:31.969304 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:31.969309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:31.969338 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:31.969692 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:31.969702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.970695 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.970700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:31.972925 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:31.972941 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:31.974756 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:31.974762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:31.974932 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:31.974964 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:35:31.975754 (+ 790us) client_negotiation.cc:240] Negotiation successful
0923 17:35:31.975892 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1391,"mutex_wait_us":5,"thread_start_us":1779,"threads_started":1}
I20260923 17:35:31.976634 29343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe671690e31487489854bb71c8657bb" candidate_uuid: "4e4fb9691c284161ad175ef91014eea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939711233b934a7692a3bd0151f4a8ec" is_pre_election: true
I20260923 17:35:31.977186 29343 raft_consensus.cc:2454] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4fb9691c284161ad175ef91014eea5 in term 0.
I20260923 17:35:31.977316 29482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe671690e31487489854bb71c8657bb" candidate_uuid: "4e4fb9691c284161ad175ef91014eea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b24ef4df9248569569bc764209741c" is_pre_election: true
I20260923 17:35:31.977730 29482 raft_consensus.cc:2454] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4fb9691c284161ad175ef91014eea5 in term 0.
I20260923 17:35:31.977845 29137 leader_election.cc:302] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4fb9691c284161ad175ef91014eea5, 939711233b934a7692a3bd0151f4a8ec; no voters:
I20260923 17:35:31.978304 29564 raft_consensus.cc:2781] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:35:31.978436 29564 raft_consensus.cc:482] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:35:31.978531 29564 raft_consensus.cc:3037] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:31.980830 29564 raft_consensus.cc:504] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } }
I20260923 17:35:31.981436 29564 leader_election.cc:288] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [CANDIDATE]: Term 1 election: Requested vote from peers b1b24ef4df9248569569bc764209741c (127.20.240.67:46455), 939711233b934a7692a3bd0151f4a8ec (127.20.240.66:45963)
I20260923 17:35:31.981905 29482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe671690e31487489854bb71c8657bb" candidate_uuid: "4e4fb9691c284161ad175ef91014eea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b24ef4df9248569569bc764209741c"
I20260923 17:35:31.981952 29343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abe671690e31487489854bb71c8657bb" candidate_uuid: "4e4fb9691c284161ad175ef91014eea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939711233b934a7692a3bd0151f4a8ec"
I20260923 17:35:31.982146 29343 raft_consensus.cc:3037] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:31.982132 29482 raft_consensus.cc:3037] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:31.985162 29482 raft_consensus.cc:2454] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4fb9691c284161ad175ef91014eea5 in term 1.
I20260923 17:35:31.985196 29343 raft_consensus.cc:2454] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4fb9691c284161ad175ef91014eea5 in term 1.
I20260923 17:35:31.985617 29136 leader_election.cc:302] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4fb9691c284161ad175ef91014eea5, b1b24ef4df9248569569bc764209741c; no voters:
I20260923 17:35:31.985946 29564 raft_consensus.cc:2781] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:31.986362 29564 raft_consensus.cc:686] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4fb9691c284161ad175ef91014eea5, State: Running, Role: LEADER
I20260923 17:35:31.986851 29564 consensus_queue.cc:237] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } }
I20260923 17:35:31.991364 29046 catalog_manager.cc:5495] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4fb9691c284161ad175ef91014eea5 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "4e4fb9691c284161ad175ef91014eea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } health_report { overall_health: UNKNOWN } } }
I20260923 17:35:32.048228 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.042675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.20.240.65:34607)
0923 17:35:32.042910 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:32.042916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:32.043229 (+ 313us) server_negotiation.cc:407] Connection header received
0923 17:35:32.043466 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:32.043470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:32.043585 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:32.043714 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:35:32.044111 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.044817 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.046139 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.046556 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.046666 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.047048 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:35:32.047639 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:35:32.047780 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:32.047887 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 17:35:32.047970 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:35:32.057027 29343 raft_consensus.cc:1261] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Refusing update from remote peer 4e4fb9691c284161ad175ef91014eea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:32.057027 29482 raft_consensus.cc:1261] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Refusing update from remote peer 4e4fb9691c284161ad175ef91014eea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:32.058176 29567 consensus_queue.cc:1035] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:32.058640 29564 consensus_queue.cc:1035] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:32.070611 29572 mvcc.cc:205] Tried to move back new op lower bound from 7332597481693999104 to 7332597481422176256. Current Snapshot: MvccSnapshot[applied={T|T < 7332597481693999104}]
I20260923 17:35:32.070921 29573 mvcc.cc:205] Tried to move back new op lower bound from 7332597481693999104 to 7332597481422176256. Current Snapshot: MvccSnapshot[applied={T|T < 7332597481693999104}]
I20260923 17:35:32.083065 29046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:32.083386 29046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:32.087387 29046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5: Sending DeleteTablet for 3 replicas of tablet abe671690e31487489854bb71c8657bb
I20260923 17:35:32.088943 29462 tablet_service.cc:1514] Processing DeleteTablet for tablet abe671690e31487489854bb71c8657bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:32 UTC) from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:42754
I20260923 17:35:32.088940 29184 tablet_service.cc:1514] Processing DeleteTablet for tablet abe671690e31487489854bb71c8657bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:32 UTC) from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:37384
I20260923 17:35:32.088927 29323 tablet_service.cc:1514] Processing DeleteTablet for tablet abe671690e31487489854bb71c8657bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:32 UTC) from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:48184
I20260923 17:35:32.091485 29046 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:35:32.092280 29046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:35:32.103699 29322 tablet_service.cc:1467] Processing CreateTablet for tablet 04972ad0b2d8418bbc17fb78912092bd (DEFAULT_TABLE table=test-table [id=b22fcf6214a24a43993eca22dd3e9908]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:32.103893 29461 tablet_service.cc:1467] Processing CreateTablet for tablet 04972ad0b2d8418bbc17fb78912092bd (DEFAULT_TABLE table=test-table [id=b22fcf6214a24a43993eca22dd3e9908]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:32.104343 29322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04972ad0b2d8418bbc17fb78912092bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:32.104388 29461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04972ad0b2d8418bbc17fb78912092bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:32.107692 29183 tablet_service.cc:1467] Processing CreateTablet for tablet 04972ad0b2d8418bbc17fb78912092bd (DEFAULT_TABLE table=test-table [id=b22fcf6214a24a43993eca22dd3e9908]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:32.108301 29183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04972ad0b2d8418bbc17fb78912092bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:32.112159 29556 tablet_bootstrap.cc:492] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec: Bootstrap starting.
I20260923 17:35:32.113812 29556 tablet_bootstrap.cc:654] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:32.119424 29557 tablet_bootstrap.cc:492] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c: Bootstrap starting.
I20260923 17:35:32.121222 29557 tablet_bootstrap.cc:654] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:32.130982 29558 tablet_bootstrap.cc:492] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5: Bootstrap starting.
I20260923 17:35:32.133152 29558 tablet_bootstrap.cc:654] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:32.145604 29557 tablet_bootstrap.cc:492] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c: No bootstrap required, opened a new log
I20260923 17:35:32.145792 29557 ts_tablet_manager.cc:1397] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.008s
I20260923 17:35:32.145969 29558 tablet_bootstrap.cc:492] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5: No bootstrap required, opened a new log
I20260923 17:35:32.146152 29558 ts_tablet_manager.cc:1397] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260923 17:35:32.146688 29557 raft_consensus.cc:348] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.146939 29557 raft_consensus.cc:374] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:32.147039 29557 raft_consensus.cc:729] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b24ef4df9248569569bc764209741c, State: Initialized, Role: FOLLOWER
I20260923 17:35:32.147010 29558 raft_consensus.cc:348] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.147279 29558 raft_consensus.cc:374] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:32.147379 29558 raft_consensus.cc:729] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4fb9691c284161ad175ef91014eea5, State: Initialized, Role: FOLLOWER
I20260923 17:35:32.147383 29557 consensus_queue.cc:260] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.147697 29558 consensus_queue.cc:260] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.149037 29558 ts_tablet_manager.cc:1428] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 17:35:32.154749 29557 ts_tablet_manager.cc:1428] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20260923 17:35:32.158483 29556 tablet_bootstrap.cc:492] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec: No bootstrap required, opened a new log
I20260923 17:35:32.158659 29556 ts_tablet_manager.cc:1397] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.011s
I20260923 17:35:32.159586 29556 raft_consensus.cc:348] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.159816 29556 raft_consensus.cc:374] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:32.159911 29556 raft_consensus.cc:729] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939711233b934a7692a3bd0151f4a8ec, State: Initialized, Role: FOLLOWER
I20260923 17:35:32.160213 29556 consensus_queue.cc:260] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.161443 29556 ts_tablet_manager.cc:1428] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 17:35:32.162875 29582 tablet_replica.cc:330] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c: stopping tablet replica
I20260923 17:35:32.163136 29580 tablet_replica.cc:330] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5: stopping tablet replica
I20260923 17:35:32.163403 29582 raft_consensus.cc:2229] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:32.163579 29580 raft_consensus.cc:2229] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:35:32.163915 29582 raft_consensus.cc:2258] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:32.164244 29580 raft_consensus.cc:2258] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:32.165058 29581 tablet_replica.cc:330] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec: stopping tablet replica
I20260923 17:35:32.165423 29581 raft_consensus.cc:2229] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:32.166276 29581 raft_consensus.cc:2258] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:32.166525 29582 ts_tablet_manager.cc:1905] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:32.166560 29580 ts_tablet_manager.cc:1905] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:32.168200 29581 ts_tablet_manager.cc:1905] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:32.173837 29582 ts_tablet_manager.cc:1918] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:32.174037 29582 log.cc:1196] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/abe671690e31487489854bb71c8657bb
I20260923 17:35:32.174567 29582 ts_tablet_manager.cc:1939] T abe671690e31487489854bb71c8657bb P b1b24ef4df9248569569bc764209741c: Deleting consensus metadata
I20260923 17:35:32.175112 29581 ts_tablet_manager.cc:1918] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:32.175299 29581 log.cc:1196] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/abe671690e31487489854bb71c8657bb
I20260923 17:35:32.175796 29581 ts_tablet_manager.cc:1939] T abe671690e31487489854bb71c8657bb P 939711233b934a7692a3bd0151f4a8ec: Deleting consensus metadata
I20260923 17:35:32.176484 29033 catalog_manager.cc:4841] TS b1b24ef4df9248569569bc764209741c (127.20.240.67:46455): tablet abe671690e31487489854bb71c8657bb (table test-table [id=3ebec32de3324ec3bb287c9966b1bc72]) successfully deleted
I20260923 17:35:32.176586 29580 ts_tablet_manager.cc:1918] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:32.176867 29580 log.cc:1196] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/abe671690e31487489854bb71c8657bb
I20260923 17:35:32.177328 29580 ts_tablet_manager.cc:1939] T abe671690e31487489854bb71c8657bb P 4e4fb9691c284161ad175ef91014eea5: Deleting consensus metadata
I20260923 17:35:32.177567 29034 catalog_manager.cc:4841] TS 939711233b934a7692a3bd0151f4a8ec (127.20.240.66:45963): tablet abe671690e31487489854bb71c8657bb (table test-table [id=3ebec32de3324ec3bb287c9966b1bc72]) successfully deleted
I20260923 17:35:32.178963 29031 catalog_manager.cc:4841] TS 4e4fb9691c284161ad175ef91014eea5 (127.20.240.65:34607): tablet abe671690e31487489854bb71c8657bb (table test-table [id=3ebec32de3324ec3bb287c9966b1bc72]) successfully deleted
I20260923 17:35:32.336103 29562 raft_consensus.cc:482] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:35:32.336308 29562 raft_consensus.cc:504] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.337260 29563 raft_consensus.cc:482] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:35:32.337507 29563 raft_consensus.cc:504] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.338181 29562 leader_election.cc:288] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4fb9691c284161ad175ef91014eea5 (127.20.240.65:34607), b1b24ef4df9248569569bc764209741c (127.20.240.67:46455)
I20260923 17:35:32.339237 29563 leader_election.cc:288] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4fb9691c284161ad175ef91014eea5 (127.20.240.65:34607), 939711233b934a7692a3bd0151f4a8ec (127.20.240.66:45963)
I20260923 17:35:32.359584 29551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.339335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:41339 (local address 127.20.240.67:46455)
0923 17:35:32.339500 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:32.339505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:32.350998 (+ 11493us) server_negotiation.cc:407] Connection header received
0923 17:35:32.351543 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:32.351550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:32.351635 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:32.351694 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:32.352605 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.353836 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.356591 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.357867 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.358000 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.358078 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:32.359104 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0923 17:35:32.359275 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":59}
I20260923 17:35:32.360397 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.338763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:46455 (local address 127.20.240.66:41339)
0923 17:35:32.350872 (+ 12109us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:32.350892 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:32.351358 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:32.351901 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:32.351906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:32.351930 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:32.352372 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.352387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.354013 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.354018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.356372 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.356388 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.358686 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.358693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.358899 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.358937 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 17:35:32.359852 (+ 915us) client_negotiation.cc:240] Negotiation successful
0923 17:35:32.359998 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11972,"mutex_wait_us":40,"thread_start_us":12245,"threads_started":1}
I20260923 17:35:32.361179 29481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04972ad0b2d8418bbc17fb78912092bd" candidate_uuid: "939711233b934a7692a3bd0151f4a8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b24ef4df9248569569bc764209741c" is_pre_election: true
I20260923 17:35:32.361546 29481 raft_consensus.cc:2454] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 939711233b934a7692a3bd0151f4a8ec in term 0.
I20260923 17:35:32.362131 29276 leader_election.cc:302] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 939711233b934a7692a3bd0151f4a8ec, b1b24ef4df9248569569bc764209741c; no voters:
I20260923 17:35:32.362568 29562 raft_consensus.cc:2781] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:35:32.362685 29562 raft_consensus.cc:482] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:35:32.362778 29562 raft_consensus.cc:3037] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:32.365374 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.350557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:54633 (local address 127.20.240.66:45963)
0923 17:35:32.351598 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:32.351604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:32.353701 (+ 2097us) server_negotiation.cc:407] Connection header received
0923 17:35:32.354013 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:32.354018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:32.354104 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:32.354176 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:32.354992 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.356261 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.362813 (+ 6552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.364165 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.364340 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.364420 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:32.364935 (+ 515us) server_negotiation.cc:299] Negotiation successful
0923 17:35:32.365093 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":928}
I20260923 17:35:32.366014 29590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.340082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:35445 (local address 127.20.240.65:34607)
0923 17:35:32.352773 (+ 12691us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:32.352778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:32.353484 (+ 706us) server_negotiation.cc:407] Connection header received
0923 17:35:32.353554 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:32.353559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:32.353628 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:32.353704 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:32.354675 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.356119 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.362153 (+ 6034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.364522 (+ 2369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.364698 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.364766 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:32.365317 (+ 551us) server_negotiation.cc:299] Negotiation successful
0923 17:35:32.365635 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12543,"thread_start_us":12357,"threads_started":1}
I20260923 17:35:32.366147 29562 raft_consensus.cc:504] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.366583 29589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.339809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:45963 (local address 127.20.240.67:54633)
0923 17:35:32.353594 (+ 13785us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:32.353614 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:32.353826 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:32.354391 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:32.354395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:32.354421 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:32.354807 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.354821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.356476 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.356480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.361974 (+ 5494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.361987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.364548 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.364555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.364734 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.364770 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:35:32.365618 (+ 848us) client_negotiation.cc:240] Negotiation successful
0923 17:35:32.365775 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13649,"thread_start_us":17162,"threads_started":1}
I20260923 17:35:32.367224 29588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.339768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34607 (local address 127.20.240.67:35445)
0923 17:35:32.353219 (+ 13451us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:32.353237 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:32.353373 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:32.353985 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:32.353989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:32.354016 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:32.354380 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.354393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.356296 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.356301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.361638 (+ 5337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.361706 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.364960 (+ 3254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.364967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.365134 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.365166 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:35:32.366498 (+ 1332us) client_negotiation.cc:240] Negotiation successful
0923 17:35:32.366836 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13318,"mutex_wait_us":2904,"thread_start_us":13160,"threads_started":1}
I20260923 17:35:32.367432 29343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04972ad0b2d8418bbc17fb78912092bd" candidate_uuid: "b1b24ef4df9248569569bc764209741c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939711233b934a7692a3bd0151f4a8ec" is_pre_election: true
I20260923 17:35:32.367671 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.338805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:49189 (local address 127.20.240.65:34607)
0923 17:35:32.339012 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:32.339019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:32.350691 (+ 11672us) server_negotiation.cc:407] Connection header received
0923 17:35:32.351702 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:32.351708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:32.351806 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:32.351864 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:35:32.357603 (+ 5739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.358634 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.360932 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.362119 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.362270 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.362375 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:32.367211 (+ 4836us) server_negotiation.cc:299] Negotiation successful
0923 17:35:32.367378 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:35:32.367733 29343 raft_consensus.cc:2379] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b1b24ef4df9248569569bc764209741c in current term 1: Already voted for candidate 939711233b934a7692a3bd0151f4a8ec in this term.
I20260923 17:35:32.368372 29481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04972ad0b2d8418bbc17fb78912092bd" candidate_uuid: "939711233b934a7692a3bd0151f4a8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b24ef4df9248569569bc764209741c"
I20260923 17:35:32.368582 29481 raft_consensus.cc:3037] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:32.368825 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.338440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34607 (local address 127.20.240.66:49189)
0923 17:35:32.350538 (+ 12098us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:32.350569 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:32.351427 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:32.352087 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:32.352091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:32.352112 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:35:32.357431 (+ 5319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.357446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.358802 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.358807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.360751 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.360766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.363380 (+ 2614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.363388 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.366959 (+ 3571us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.366998 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 17:35:32.368305 (+ 1307us) client_negotiation.cc:240] Negotiation successful
0923 17:35:32.368460 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11911,"mutex_wait_us":3319,"thread_start_us":11718,"threads_started":1}
I20260923 17:35:32.368820 29209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04972ad0b2d8418bbc17fb78912092bd" candidate_uuid: "b1b24ef4df9248569569bc764209741c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4fb9691c284161ad175ef91014eea5" is_pre_election: true
I20260923 17:35:32.369307 29562 leader_election.cc:288] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [CANDIDATE]: Term 1 election: Requested vote from peers 4e4fb9691c284161ad175ef91014eea5 (127.20.240.65:34607), b1b24ef4df9248569569bc764209741c (127.20.240.67:46455)
I20260923 17:35:32.369432 29209 raft_consensus.cc:2454] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1b24ef4df9248569569bc764209741c in term 0.
I20260923 17:35:32.370241 29208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04972ad0b2d8418bbc17fb78912092bd" candidate_uuid: "939711233b934a7692a3bd0151f4a8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4fb9691c284161ad175ef91014eea5"
I20260923 17:35:32.370312 29413 leader_election.cc:302] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4fb9691c284161ad175ef91014eea5, b1b24ef4df9248569569bc764209741c; no voters:
I20260923 17:35:32.370536 29208 raft_consensus.cc:3037] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:32.371029 29209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04972ad0b2d8418bbc17fb78912092bd" candidate_uuid: "939711233b934a7692a3bd0151f4a8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4fb9691c284161ad175ef91014eea5" is_pre_election: true
I20260923 17:35:32.371907 29481 raft_consensus.cc:2454] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 939711233b934a7692a3bd0151f4a8ec in term 1.
I20260923 17:35:32.372066 29563 raft_consensus.cc:2741] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260923 17:35:32.372893 29276 leader_election.cc:302] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 939711233b934a7692a3bd0151f4a8ec, b1b24ef4df9248569569bc764209741c; no voters:
I20260923 17:35:32.373216 29562 raft_consensus.cc:2781] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:32.373725 29562 raft_consensus.cc:686] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [term 1 LEADER]: Becoming Leader. State: Replica: 939711233b934a7692a3bd0151f4a8ec, State: Running, Role: LEADER
I20260923 17:35:32.374171 29562 consensus_queue.cc:237] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.374536 29208 raft_consensus.cc:2454] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 939711233b934a7692a3bd0151f4a8ec in term 1.
I20260923 17:35:32.378301 29046 catalog_manager.cc:5495] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 939711233b934a7692a3bd0151f4a8ec (127.20.240.66). New cstate: current_term: 1 leader_uuid: "939711233b934a7692a3bd0151f4a8ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } health_report { overall_health: UNKNOWN } } }
Sep 23 17:35:32 dist-test-slave-ppjl krb5kdc[29002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184929, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.126@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
I20260923 17:35:32.461773 29599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.441808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48938 (local address 127.20.240.126:43579)
0923 17:35:32.442790 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:32.442795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:32.442815 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:35:32.442866 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:32.442869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:32.442932 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:32.443045 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:32.444185 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.444956 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.446153 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.446543 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.446650 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.455369 (+ 8719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:32.455375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:32.455379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:32.455413 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:32.458347 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:32.459418 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:32.459423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:32.459425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:32.459569 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:32.460089 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:32.460093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:32.460095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:32.460397 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:32.460524 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:32.461133 (+ 609us) server_negotiation.cc:299] Negotiation successful
0923 17:35:32.461426 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":836,"thread_start_us":487,"threads_started":1}
I20260923 17:35:32.461786 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.441729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:43579 (local address 127.0.0.1:48938)
0923 17:35:32.442470 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:32.442495 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:32.442661 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:32.443098 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:32.443108 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:32.443777 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:32.444038 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.444050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.445078 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.445083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.446025 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.446037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.446705 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.446711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.446860 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.452442 (+ 5582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:32.452464 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:32.455006 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:32.458535 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:32.458555 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:32.458565 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:32.459289 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:32.459698 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:32.459703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:32.459707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:32.459918 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:32.460541 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:32.460550 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:32.460775 (+ 225us) client_negotiation.cc:769] Sending connection context
0923 17:35:32.460975 (+ 200us) client_negotiation.cc:240] Negotiation successful
0923 17:35:32.461301 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":415,"threads_started":1}
I20260923 17:35:32.466980 29046 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:48938:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 17:35:32.468103 29046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:35:32.476823 29323 tablet_service.cc:1467] Processing CreateTablet for tablet 9793d7e9da2d44958764366b9a1fc170 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=889c6003b53c470b84ff964a29f47cfa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:35:32.477432 29323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9793d7e9da2d44958764366b9a1fc170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:32.479957 29184 tablet_service.cc:1467] Processing CreateTablet for tablet 9793d7e9da2d44958764366b9a1fc170 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=889c6003b53c470b84ff964a29f47cfa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:35:32.480522 29184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9793d7e9da2d44958764366b9a1fc170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:32.484381 29462 tablet_service.cc:1467] Processing CreateTablet for tablet 9793d7e9da2d44958764366b9a1fc170 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=889c6003b53c470b84ff964a29f47cfa]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:35:32.484961 29462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9793d7e9da2d44958764366b9a1fc170. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:32.495293 29557 tablet_bootstrap.cc:492] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c: Bootstrap starting.
I20260923 17:35:32.498606 29557 tablet_bootstrap.cc:654] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:32.500900 29558 tablet_bootstrap.cc:492] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5: Bootstrap starting.
I20260923 17:35:32.502517 29558 tablet_bootstrap.cc:654] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:32.504861 29557 tablet_bootstrap.cc:492] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c: No bootstrap required, opened a new log
I20260923 17:35:32.505096 29557 ts_tablet_manager.cc:1397] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260923 17:35:32.505888 29557 raft_consensus.cc:348] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.506253 29557 raft_consensus.cc:374] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:32.506363 29557 raft_consensus.cc:729] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b24ef4df9248569569bc764209741c, State: Initialized, Role: FOLLOWER
I20260923 17:35:32.506682 29557 consensus_queue.cc:260] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.506966 29556 tablet_bootstrap.cc:492] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec: Bootstrap starting.
I20260923 17:35:32.507200 29563 tablet_replica.cc:439] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } }
I20260923 17:35:32.507433 29563 tablet_replica.cc:442] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:35:32.508494 29557 ts_tablet_manager.cc:1428] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260923 17:35:32.508844 29556 tablet_bootstrap.cc:654] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:32.512467 29556 tablet_bootstrap.cc:492] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec: No bootstrap required, opened a new log
I20260923 17:35:32.512641 29556 ts_tablet_manager.cc:1397] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260923 17:35:32.513403 29556 raft_consensus.cc:348] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.513640 29556 raft_consensus.cc:374] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:32.513720 29556 raft_consensus.cc:729] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939711233b934a7692a3bd0151f4a8ec, State: Initialized, Role: FOLLOWER
I20260923 17:35:32.513986 29556 consensus_queue.cc:260] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.514415 29562 tablet_replica.cc:439] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } }
I20260923 17:35:32.514626 29558 tablet_bootstrap.cc:492] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5: No bootstrap required, opened a new log
I20260923 17:35:32.514688 29562 tablet_replica.cc:442] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:35:32.514801 29558 ts_tablet_manager.cc:1397] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260923 17:35:32.515636 29556 ts_tablet_manager.cc:1428] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260923 17:35:32.515583 29558 raft_consensus.cc:348] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.515836 29558 raft_consensus.cc:374] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:32.515933 29558 raft_consensus.cc:729] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4fb9691c284161ad175ef91014eea5, State: Initialized, Role: FOLLOWER
I20260923 17:35:32.516175 29558 consensus_queue.cc:260] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.516705 29558 ts_tablet_manager.cc:1428] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:35:32.516618 29564 tablet_replica.cc:439] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } }
I20260923 17:35:32.516886 29564 tablet_replica.cc:442] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:35:32.789070 29564 raft_consensus.cc:482] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:35:32.789273 29564 raft_consensus.cc:504] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.789932 29564 leader_election.cc:288] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 939711233b934a7692a3bd0151f4a8ec (127.20.240.66:45963), b1b24ef4df9248569569bc764209741c (127.20.240.67:46455)
I20260923 17:35:32.790522 29343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9793d7e9da2d44958764366b9a1fc170" candidate_uuid: "4e4fb9691c284161ad175ef91014eea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939711233b934a7692a3bd0151f4a8ec" is_pre_election: true
I20260923 17:35:32.790585 29481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9793d7e9da2d44958764366b9a1fc170" candidate_uuid: "4e4fb9691c284161ad175ef91014eea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b24ef4df9248569569bc764209741c" is_pre_election: true
I20260923 17:35:32.790764 29343 raft_consensus.cc:2454] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4fb9691c284161ad175ef91014eea5 in term 0.
I20260923 17:35:32.790824 29481 raft_consensus.cc:2454] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4fb9691c284161ad175ef91014eea5 in term 0.
I20260923 17:35:32.791208 29137 leader_election.cc:302] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4fb9691c284161ad175ef91014eea5, 939711233b934a7692a3bd0151f4a8ec; no voters:
I20260923 17:35:32.791502 29564 raft_consensus.cc:2781] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:35:32.791616 29564 raft_consensus.cc:482] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:35:32.791706 29564 raft_consensus.cc:3037] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:32.793654 29564 raft_consensus.cc:504] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.794222 29564 leader_election.cc:288] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [CANDIDATE]: Term 1 election: Requested vote from peers 939711233b934a7692a3bd0151f4a8ec (127.20.240.66:45963), b1b24ef4df9248569569bc764209741c (127.20.240.67:46455)
I20260923 17:35:32.794773 29481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9793d7e9da2d44958764366b9a1fc170" candidate_uuid: "4e4fb9691c284161ad175ef91014eea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b24ef4df9248569569bc764209741c"
I20260923 17:35:32.794791 29343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9793d7e9da2d44958764366b9a1fc170" candidate_uuid: "4e4fb9691c284161ad175ef91014eea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939711233b934a7692a3bd0151f4a8ec"
I20260923 17:35:32.794986 29343 raft_consensus.cc:3037] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:32.794994 29481 raft_consensus.cc:3037] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:32.797115 29343 raft_consensus.cc:2454] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4fb9691c284161ad175ef91014eea5 in term 1.
I20260923 17:35:32.797215 29481 raft_consensus.cc:2454] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4fb9691c284161ad175ef91014eea5 in term 1.
I20260923 17:35:32.797636 29137 leader_election.cc:302] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4fb9691c284161ad175ef91014eea5, 939711233b934a7692a3bd0151f4a8ec; no voters:
I20260923 17:35:32.797967 29564 raft_consensus.cc:2781] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:32.798317 29564 raft_consensus.cc:686] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4fb9691c284161ad175ef91014eea5, State: Running, Role: LEADER
I20260923 17:35:32.798579 29564 consensus_queue.cc:237] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:32.800103 29603 tablet_replica.cc:439] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5: TxnStatusTablet state changed. Reason: New leader 4e4fb9691c284161ad175ef91014eea5. Latest consensus state: current_term: 1 leader_uuid: "4e4fb9691c284161ad175ef91014eea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } }
I20260923 17:35:32.800364 29603 tablet_replica.cc:442] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:35:32.800943 29605 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 17:35:32.801206 29605 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 17:35:32.801779 29047 catalog_manager.cc:5495] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4fb9691c284161ad175ef91014eea5 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "4e4fb9691c284161ad175ef91014eea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } health_report { overall_health: UNKNOWN } } }
I20260923 17:35:32.814214 29562 consensus_queue.cc:1035] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:32.822263 29562 consensus_queue.cc:1035] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:32.887584 29598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.882331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34607 (local address 127.0.0.1:37402)
0923 17:35:32.882562 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:32.882579 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:32.882726 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:32.883342 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:32.883347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:32.883359 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:35:32.883551 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.883563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.884539 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.884545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.885600 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.885611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.886317 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.886323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.886494 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.886556 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:35:32.886930 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:35:32.886952 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 17:35:32.887035 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 17:35:32.887239 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 17:35:32.887688 29614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.882485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.20.240.65:34607)
0923 17:35:32.883016 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:32.883021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:32.883035 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:35:32.883093 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:32.883096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:32.883203 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:32.883291 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:35:32.883679 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.884413 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.885729 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.886167 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.886275 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.886728 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:35:32.886837 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:35:32.886912 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:32.887099 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 17:35:32.887198 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":252,"threads_started":1}
I20260923 17:35:32.891471 29481 raft_consensus.cc:1261] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Refusing update from remote peer 4e4fb9691c284161ad175ef91014eea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:32.891520 29343 raft_consensus.cc:1261] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Refusing update from remote peer 4e4fb9691c284161ad175ef91014eea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:32.892098 29603 consensus_queue.cc:1035] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:32.892409 29564 consensus_queue.cc:1035] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:32.895174 29591 tablet_replica.cc:439] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec: TxnStatusTablet state changed. Reason: New leader 4e4fb9691c284161ad175ef91014eea5. Latest consensus state: current_term: 1 leader_uuid: "4e4fb9691c284161ad175ef91014eea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } }
I20260923 17:35:32.895488 29591 tablet_replica.cc:442] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:35:32.899431 29603 tablet_replica.cc:439] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e4fb9691c284161ad175ef91014eea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } }
I20260923 17:35:32.899713 29603 tablet_replica.cc:442] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:35:32.904263 29591 tablet_replica.cc:439] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e4fb9691c284161ad175ef91014eea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } }
I20260923 17:35:32.904531 29591 tablet_replica.cc:442] T 9793d7e9da2d44958764366b9a1fc170 P 939711233b934a7692a3bd0151f4a8ec: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:35:32.908195 29563 tablet_replica.cc:439] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c: TxnStatusTablet state changed. Reason: New leader 4e4fb9691c284161ad175ef91014eea5. Latest consensus state: current_term: 1 leader_uuid: "4e4fb9691c284161ad175ef91014eea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } }
I20260923 17:35:32.908545 29563 tablet_replica.cc:442] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:35:32.911732 29564 tablet_replica.cc:439] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e4fb9691c284161ad175ef91014eea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } }
I20260923 17:35:32.912060 29564 tablet_replica.cc:442] T 9793d7e9da2d44958764366b9a1fc170 P 4e4fb9691c284161ad175ef91014eea5: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:35:32.919692 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.910170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.20.240.66:45963)
0923 17:35:32.911487 (+ 1317us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:32.911493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:32.911512 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:35:32.911621 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:32.911626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:32.911746 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:32.911933 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:35:32.912766 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.913962 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.916192 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.916837 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.917042 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.917668 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:35:32.918610 (+ 942us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:35:32.918743 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:32.919273 (+ 530us) server_negotiation.cc:299] Negotiation successful
0923 17:35:32.919359 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1188,"thread_start_us":1070,"threads_started":1}
I20260923 17:35:32.920918 29563 tablet_replica.cc:439] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e4fb9691c284161ad175ef91014eea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } } }
I20260923 17:35:32.921196 29563 tablet_replica.cc:442] T 9793d7e9da2d44958764366b9a1fc170 P b1b24ef4df9248569569bc764209741c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:35:32.936439 29626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.930262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34607 (local address 127.20.240.66:38475)
0923 17:35:32.930831 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:32.930848 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:32.930972 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:32.931325 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:32.931329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:32.931339 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:35:32.931593 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.931601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.933095 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.933100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.934273 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.934288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.935285 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.935291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.935425 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.935492 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:35:32.935941 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:35:32.935958 (+ 17us) client_negotiation.cc:769] Sending connection context
0923 17:35:32.936026 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 17:35:32.936145 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":275,"threads_started":1}
I20260923 17:35:32.936443 29614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.930398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:38475 (local address 127.20.240.65:34607)
0923 17:35:32.930605 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:32.930611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:32.930929 (+ 318us) server_negotiation.cc:407] Connection header received
0923 17:35:32.931090 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:32.931096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:32.931211 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:32.931300 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:35:32.931771 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.932908 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.934435 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.935080 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.935237 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.935691 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:35:32.935836 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:35:32.935929 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:32.936075 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 17:35:32.936162 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 17:35:32.976437 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.968828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:56649 (local address 127.20.240.66:45963)
0923 17:35:32.969014 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:32.969021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:32.969557 (+ 536us) server_negotiation.cc:407] Connection header received
0923 17:35:32.969753 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:32.969759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:32.969838 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:32.969920 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:35:32.970575 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.971828 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.974255 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.974901 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.975020 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.975659 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:35:32.975789 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:35:32.975846 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:32.976065 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 17:35:32.976153 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 17:35:32.976629 29628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:32.968780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:45963 (local address 127.20.240.65:56649)
0923 17:35:32.969434 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:32.969456 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:32.969608 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:32.969954 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:32.969959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:32.969972 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:35:32.970364 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.970376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.971973 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.971979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.974110 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:32.974126 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:32.975197 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:32.975205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:32.975376 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:32.975460 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:35:32.975923 (+ 463us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:35:32.975941 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 17:35:32.976020 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 17:35:32.976170 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":328,"threads_started":1}
I20260923 17:35:33.022184 29045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:33.022490 29045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:33.026471 29045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5: Sending DeleteTablet for 3 replicas of tablet 04972ad0b2d8418bbc17fb78912092bd
I20260923 17:35:33.027544 29323 tablet_service.cc:1514] Processing DeleteTablet for tablet 04972ad0b2d8418bbc17fb78912092bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:33 UTC) from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:48184
I20260923 17:35:33.027976 29462 tablet_service.cc:1514] Processing DeleteTablet for tablet 04972ad0b2d8418bbc17fb78912092bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:33 UTC) from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:42754
I20260923 17:35:33.028399 29184 tablet_service.cc:1514] Processing DeleteTablet for tablet 04972ad0b2d8418bbc17fb78912092bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:33 UTC) from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:37384
I20260923 17:35:33.052578 29631 tablet_replica.cc:330] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec: stopping tablet replica
I20260923 17:35:33.052934 29633 tablet_replica.cc:330] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5: stopping tablet replica
I20260923 17:35:33.052989 29631 raft_consensus.cc:2229] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:35:33.053176 29633 raft_consensus.cc:2229] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:33.053471 29633 raft_consensus.cc:2258] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:33.053892 29631 raft_consensus.cc:2258] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:33.055105 29631 ts_tablet_manager.cc:1905] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:33.059994 29633 ts_tablet_manager.cc:1905] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:33.062964 29631 ts_tablet_manager.cc:1918] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:35:33.063266 29631 log.cc:1196] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/04972ad0b2d8418bbc17fb78912092bd
I20260923 17:35:33.063558 29632 tablet_replica.cc:330] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c: stopping tablet replica
I20260923 17:35:33.063779 29632 raft_consensus.cc:2229] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:33.063970 29631 ts_tablet_manager.cc:1939] T 04972ad0b2d8418bbc17fb78912092bd P 939711233b934a7692a3bd0151f4a8ec: Deleting consensus metadata
I20260923 17:35:33.064043 29632 raft_consensus.cc:2258] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:33.065258 29632 ts_tablet_manager.cc:1905] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:33.065233 29034 catalog_manager.cc:4841] TS 939711233b934a7692a3bd0151f4a8ec (127.20.240.66:45963): tablet 04972ad0b2d8418bbc17fb78912092bd (table test-table [id=b22fcf6214a24a43993eca22dd3e9908]) successfully deleted
I20260923 17:35:33.072304 29632 ts_tablet_manager.cc:1918] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:35:33.072464 29632 log.cc:1196] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/04972ad0b2d8418bbc17fb78912092bd
I20260923 17:35:33.072892 29632 ts_tablet_manager.cc:1939] T 04972ad0b2d8418bbc17fb78912092bd P b1b24ef4df9248569569bc764209741c: Deleting consensus metadata
I20260923 17:35:33.074093 29033 catalog_manager.cc:4841] TS b1b24ef4df9248569569bc764209741c (127.20.240.67:46455): tablet 04972ad0b2d8418bbc17fb78912092bd (table test-table [id=b22fcf6214a24a43993eca22dd3e9908]) successfully deleted
I20260923 17:35:33.076713 29633 ts_tablet_manager.cc:1918] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:35:33.076941 29633 log.cc:1196] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/04972ad0b2d8418bbc17fb78912092bd
I20260923 17:35:33.077445 29633 ts_tablet_manager.cc:1939] T 04972ad0b2d8418bbc17fb78912092bd P 4e4fb9691c284161ad175ef91014eea5: Deleting consensus metadata
I20260923 17:35:33.078646 29031 catalog_manager.cc:4841] TS 4e4fb9691c284161ad175ef91014eea5 (127.20.240.65:34607): tablet 04972ad0b2d8418bbc17fb78912092bd (table test-table [id=b22fcf6214a24a43993eca22dd3e9908]) successfully deleted
Sep 23 17:35:33 dist-test-slave-ppjl krb5kdc[29002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:35:33 dist-test-slave-ppjl krb5kdc[29002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:33 dist-test-slave-ppjl krb5kdc[29002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:35:33.114078 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:33.091449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48950 (local address 127.20.240.126:43579)
0923 17:35:33.104027 (+ 12578us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:33.104036 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:33.104654 (+ 618us) server_negotiation.cc:407] Connection header received
0923 17:35:33.104836 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:33.104841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:33.104914 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:33.105006 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:33.106396 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:33.107325 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:33.108733 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:33.109124 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:33.109233 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:33.113832 (+ 4599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48950: BlockingRecv error: recv got EOF from 127.0.0.1:48950 (error 108)
Metrics: {"server-negotiator.queue_time_us":12402,"thread_start_us":12595,"threads_started":1}
Sep 23 17:35:33 dist-test-slave-ppjl krb5kdc[29002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM
I20260923 17:35:33.134981 29641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:33.120473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48960 (local address 127.20.240.126:43579)
0923 17:35:33.120701 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:33.120708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:33.121203 (+ 495us) server_negotiation.cc:407] Connection header received
0923 17:35:33.121376 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:33.121382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:33.121449 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:33.121531 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:33.122654 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:33.123477 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:33.124753 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:33.125154 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:33.125262 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:33.129645 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:33.129651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:33.129656 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:33.129703 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:33.132460 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:33.133105 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:33.133109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:33.133111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:33.133243 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:33.133604 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:33.133608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:33.133610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:33.133888 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:33.134128 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:33.134437 (+ 309us) server_negotiation.cc:299] Negotiation successful
0923 17:35:33.134652 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:35:33.138511 29045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:35:33.139240 29045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:35:33.146246 29184 tablet_service.cc:1467] Processing CreateTablet for tablet 249873a166aa4d2faa02a964b16ec3a7 (DEFAULT_TABLE table=test-table [id=7aedcca0c27c4a539f810868db35cc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:33.146452 29323 tablet_service.cc:1467] Processing CreateTablet for tablet 249873a166aa4d2faa02a964b16ec3a7 (DEFAULT_TABLE table=test-table [id=7aedcca0c27c4a539f810868db35cc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:33.146760 29184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249873a166aa4d2faa02a964b16ec3a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:33.146864 29323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249873a166aa4d2faa02a964b16ec3a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:33.151836 29462 tablet_service.cc:1467] Processing CreateTablet for tablet 249873a166aa4d2faa02a964b16ec3a7 (DEFAULT_TABLE table=test-table [id=7aedcca0c27c4a539f810868db35cc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:33.152248 29462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 249873a166aa4d2faa02a964b16ec3a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:33.161618 29647 tablet_bootstrap.cc:492] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec: Bootstrap starting.
I20260923 17:35:33.161726 29648 tablet_bootstrap.cc:492] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5: Bootstrap starting.
I20260923 17:35:33.164361 29648 tablet_bootstrap.cc:654] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:33.164848 29647 tablet_bootstrap.cc:654] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:33.168047 29649 tablet_bootstrap.cc:492] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c: Bootstrap starting.
I20260923 17:35:33.169770 29649 tablet_bootstrap.cc:654] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:33.173766 29648 tablet_bootstrap.cc:492] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5: No bootstrap required, opened a new log
I20260923 17:35:33.173954 29648 ts_tablet_manager.cc:1397] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260923 17:35:33.174846 29648 raft_consensus.cc:348] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.175129 29648 raft_consensus.cc:374] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:33.175259 29648 raft_consensus.cc:729] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4fb9691c284161ad175ef91014eea5, State: Initialized, Role: FOLLOWER
I20260923 17:35:33.175534 29648 consensus_queue.cc:260] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.176158 29647 tablet_bootstrap.cc:492] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec: No bootstrap required, opened a new log
I20260923 17:35:33.176358 29647 ts_tablet_manager.cc:1397] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260923 17:35:33.177246 29647 raft_consensus.cc:348] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.177564 29647 raft_consensus.cc:374] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:33.177712 29647 raft_consensus.cc:729] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939711233b934a7692a3bd0151f4a8ec, State: Initialized, Role: FOLLOWER
I20260923 17:35:33.178050 29647 consensus_queue.cc:260] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.178498 29648 ts_tablet_manager.cc:1428] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260923 17:35:33.179807 29647 ts_tablet_manager.cc:1428] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260923 17:35:33.183439 29649 tablet_bootstrap.cc:492] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c: No bootstrap required, opened a new log
I20260923 17:35:33.183604 29649 ts_tablet_manager.cc:1397] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260923 17:35:33.184336 29649 raft_consensus.cc:348] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.184551 29649 raft_consensus.cc:374] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:33.184651 29649 raft_consensus.cc:729] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b24ef4df9248569569bc764209741c, State: Initialized, Role: FOLLOWER
I20260923 17:35:33.184880 29649 consensus_queue.cc:260] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.185398 29649 ts_tablet_manager.cc:1428] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260923 17:35:33.193182 29563 raft_consensus.cc:482] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:35:33.193358 29563 raft_consensus.cc:504] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.194037 29563 leader_election.cc:288] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e4fb9691c284161ad175ef91014eea5 (127.20.240.65:34607), 939711233b934a7692a3bd0151f4a8ec (127.20.240.66:45963)
I20260923 17:35:33.194681 29343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249873a166aa4d2faa02a964b16ec3a7" candidate_uuid: "b1b24ef4df9248569569bc764209741c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939711233b934a7692a3bd0151f4a8ec" is_pre_election: true
I20260923 17:35:33.194629 29208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249873a166aa4d2faa02a964b16ec3a7" candidate_uuid: "b1b24ef4df9248569569bc764209741c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4fb9691c284161ad175ef91014eea5" is_pre_election: true
I20260923 17:35:33.194957 29208 raft_consensus.cc:2454] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1b24ef4df9248569569bc764209741c in term 0.
I20260923 17:35:33.195065 29343 raft_consensus.cc:2454] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1b24ef4df9248569569bc764209741c in term 0.
I20260923 17:35:33.195657 29413 leader_election.cc:302] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4fb9691c284161ad175ef91014eea5, b1b24ef4df9248569569bc764209741c; no voters:
I20260923 17:35:33.195943 29563 raft_consensus.cc:2781] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:35:33.196086 29563 raft_consensus.cc:482] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:35:33.196203 29563 raft_consensus.cc:3037] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:33.198699 29563 raft_consensus.cc:504] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.199884 29208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249873a166aa4d2faa02a964b16ec3a7" candidate_uuid: "b1b24ef4df9248569569bc764209741c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e4fb9691c284161ad175ef91014eea5"
I20260923 17:35:33.199903 29343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "249873a166aa4d2faa02a964b16ec3a7" candidate_uuid: "b1b24ef4df9248569569bc764209741c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939711233b934a7692a3bd0151f4a8ec"
I20260923 17:35:33.200147 29343 raft_consensus.cc:3037] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:33.200160 29208 raft_consensus.cc:3037] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:33.203639 29343 raft_consensus.cc:2454] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1b24ef4df9248569569bc764209741c in term 1.
I20260923 17:35:33.203670 29208 raft_consensus.cc:2454] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1b24ef4df9248569569bc764209741c in term 1.
I20260923 17:35:33.204258 29416 leader_election.cc:302] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4e4fb9691c284161ad175ef91014eea5, 939711233b934a7692a3bd0151f4a8ec, b1b24ef4df9248569569bc764209741c; no voters:
I20260923 17:35:33.204470 29563 leader_election.cc:288] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [CANDIDATE]: Term 1 election: Requested vote from peers 4e4fb9691c284161ad175ef91014eea5 (127.20.240.65:34607), 939711233b934a7692a3bd0151f4a8ec (127.20.240.66:45963)
I20260923 17:35:33.204748 29563 raft_consensus.cc:2781] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:33.204977 29563 raft_consensus.cc:686] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [term 1 LEADER]: Becoming Leader. State: Replica: b1b24ef4df9248569569bc764209741c, State: Running, Role: LEADER
I20260923 17:35:33.205511 29563 consensus_queue.cc:237] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.221331 29047 catalog_manager.cc:5495] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c reported cstate change: term changed from 0 to 1, leader changed from <none> to b1b24ef4df9248569569bc764209741c (127.20.240.67). New cstate: current_term: 1 leader_uuid: "b1b24ef4df9248569569bc764209741c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } health_report { overall_health: HEALTHY } } }
I20260923 17:35:33.249897 29656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:33.244031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.20.240.67:46455)
0923 17:35:33.244626 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:33.244631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:33.244650 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:35:33.244773 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:33.244779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:33.244872 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:33.245024 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:35:33.245422 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:33.246184 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:33.247548 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:33.247972 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:33.248083 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:33.248523 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:35:33.249199 (+ 676us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:35:33.249329 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:33.249424 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 17:35:33.249504 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":331,"threads_started":1}
I20260923 17:35:33.253795 29208 raft_consensus.cc:1261] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5 [term 1 FOLLOWER]: Refusing update from remote peer b1b24ef4df9248569569bc764209741c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:33.253773 29343 raft_consensus.cc:1261] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Refusing update from remote peer b1b24ef4df9248569569bc764209741c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:33.254699 29563 consensus_queue.cc:1035] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [LEADER]: Connected to new peer: Peer: permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:33.255422 29563 consensus_queue.cc:1035] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:33.297009 29045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:33.297292 29045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:33.301236 29045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5: Sending DeleteTablet for 3 replicas of tablet 249873a166aa4d2faa02a964b16ec3a7
I20260923 17:35:33.303108 29184 tablet_service.cc:1514] Processing DeleteTablet for tablet 249873a166aa4d2faa02a964b16ec3a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:33 UTC) from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:37384
I20260923 17:35:33.303454 29633 tablet_replica.cc:330] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5: stopping tablet replica
I20260923 17:35:33.303534 29323 tablet_service.cc:1514] Processing DeleteTablet for tablet 249873a166aa4d2faa02a964b16ec3a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:33 UTC) from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:48184
I20260923 17:35:33.303664 29633 raft_consensus.cc:2229] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:33.303916 29633 raft_consensus.cc:2258] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:33.304224 29631 tablet_replica.cc:330] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec: stopping tablet replica
I20260923 17:35:33.304139 29462 tablet_service.cc:1514] Processing DeleteTablet for tablet 249873a166aa4d2faa02a964b16ec3a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:33 UTC) from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:42754
I20260923 17:35:33.304435 29631 raft_consensus.cc:2229] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:33.304661 29632 tablet_replica.cc:330] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c: stopping tablet replica
I20260923 17:35:33.304685 29631 raft_consensus.cc:2258] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:33.304859 29632 raft_consensus.cc:2229] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:35:33.305358 29632 raft_consensus.cc:2258] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:33.306689 29633 ts_tablet_manager.cc:1905] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:33.307715 29632 ts_tablet_manager.cc:1905] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:33.311178 29631 ts_tablet_manager.cc:1905] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:33.314342 29632 ts_tablet_manager.cc:1918] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:33.314649 29632 log.cc:1196] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/249873a166aa4d2faa02a964b16ec3a7
I20260923 17:35:33.315181 29632 ts_tablet_manager.cc:1939] T 249873a166aa4d2faa02a964b16ec3a7 P b1b24ef4df9248569569bc764209741c: Deleting consensus metadata
I20260923 17:35:33.315675 29045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:35:33.316435 29045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:35:33.316586 29033 catalog_manager.cc:4841] TS b1b24ef4df9248569569bc764209741c (127.20.240.67:46455): tablet 249873a166aa4d2faa02a964b16ec3a7 (table test-table [id=7aedcca0c27c4a539f810868db35cc3d]) successfully deleted
I20260923 17:35:33.321293 29631 ts_tablet_manager.cc:1918] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:33.321451 29631 log.cc:1196] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/249873a166aa4d2faa02a964b16ec3a7
I20260923 17:35:33.326570 29633 ts_tablet_manager.cc:1918] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:35:33.326756 29633 log.cc:1196] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/249873a166aa4d2faa02a964b16ec3a7
I20260923 17:35:33.327626 29184 tablet_service.cc:1467] Processing CreateTablet for tablet 1b991de684684727838b5b8b3e63dd23 (DEFAULT_TABLE table=test-table [id=aa6b3be747e746d48670b545dd17bf52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:33.328007 29631 ts_tablet_manager.cc:1939] T 249873a166aa4d2faa02a964b16ec3a7 P 939711233b934a7692a3bd0151f4a8ec: Deleting consensus metadata
I20260923 17:35:33.328044 29184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b991de684684727838b5b8b3e63dd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:33.329345 29633 ts_tablet_manager.cc:1939] T 249873a166aa4d2faa02a964b16ec3a7 P 4e4fb9691c284161ad175ef91014eea5: Deleting consensus metadata
I20260923 17:35:33.330741 29323 tablet_service.cc:1467] Processing CreateTablet for tablet 1b991de684684727838b5b8b3e63dd23 (DEFAULT_TABLE table=test-table [id=aa6b3be747e746d48670b545dd17bf52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:33.331148 29323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b991de684684727838b5b8b3e63dd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:33.333995 29031 catalog_manager.cc:4841] TS 4e4fb9691c284161ad175ef91014eea5 (127.20.240.65:34607): tablet 249873a166aa4d2faa02a964b16ec3a7 (table test-table [id=7aedcca0c27c4a539f810868db35cc3d]) successfully deleted
I20260923 17:35:33.334372 29648 tablet_bootstrap.cc:492] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5: Bootstrap starting.
I20260923 17:35:33.335937 29648 tablet_bootstrap.cc:654] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:33.335920 29462 tablet_service.cc:1467] Processing CreateTablet for tablet 1b991de684684727838b5b8b3e63dd23 (DEFAULT_TABLE table=test-table [id=aa6b3be747e746d48670b545dd17bf52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:35:33.336524 29462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b991de684684727838b5b8b3e63dd23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:33.337598 29647 tablet_bootstrap.cc:492] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec: Bootstrap starting.
I20260923 17:35:33.339291 29647 tablet_bootstrap.cc:654] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:33.339324 29034 catalog_manager.cc:4841] TS 939711233b934a7692a3bd0151f4a8ec (127.20.240.66:45963): tablet 249873a166aa4d2faa02a964b16ec3a7 (table test-table [id=7aedcca0c27c4a539f810868db35cc3d]) successfully deleted
I20260923 17:35:33.344422 29649 tablet_bootstrap.cc:492] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c: Bootstrap starting.
I20260923 17:35:33.346259 29649 tablet_bootstrap.cc:654] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:33.356015 29649 tablet_bootstrap.cc:492] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c: No bootstrap required, opened a new log
I20260923 17:35:33.356242 29649 ts_tablet_manager.cc:1397] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260923 17:35:33.357191 29649 raft_consensus.cc:348] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.357491 29649 raft_consensus.cc:374] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:33.357587 29649 raft_consensus.cc:729] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b24ef4df9248569569bc764209741c, State: Initialized, Role: FOLLOWER
I20260923 17:35:33.357880 29649 consensus_queue.cc:260] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.358506 29649 ts_tablet_manager.cc:1428] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260923 17:35:33.360565 29647 tablet_bootstrap.cc:492] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec: No bootstrap required, opened a new log
I20260923 17:35:33.360797 29647 ts_tablet_manager.cc:1397] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.005s
I20260923 17:35:33.361687 29647 raft_consensus.cc:348] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.361970 29647 raft_consensus.cc:374] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:33.362073 29647 raft_consensus.cc:729] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939711233b934a7692a3bd0151f4a8ec, State: Initialized, Role: FOLLOWER
I20260923 17:35:33.362326 29647 consensus_queue.cc:260] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.365146 29647 ts_tablet_manager.cc:1428] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260923 17:35:33.372526 29648 tablet_bootstrap.cc:492] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5: No bootstrap required, opened a new log
I20260923 17:35:33.372732 29648 ts_tablet_manager.cc:1397] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.005s
I20260923 17:35:33.373498 29648 raft_consensus.cc:348] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.373732 29648 raft_consensus.cc:374] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:33.373819 29648 raft_consensus.cc:729] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4fb9691c284161ad175ef91014eea5, State: Initialized, Role: FOLLOWER
I20260923 17:35:33.374085 29648 consensus_queue.cc:260] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.375310 29648 ts_tablet_manager.cc:1428] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260923 17:35:33.789237 29603 raft_consensus.cc:482] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:35:33.789459 29603 raft_consensus.cc:504] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.790104 29603 leader_election.cc:288] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 939711233b934a7692a3bd0151f4a8ec (127.20.240.66:45963), b1b24ef4df9248569569bc764209741c (127.20.240.67:46455)
I20260923 17:35:33.790735 29343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b991de684684727838b5b8b3e63dd23" candidate_uuid: "4e4fb9691c284161ad175ef91014eea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939711233b934a7692a3bd0151f4a8ec" is_pre_election: true
I20260923 17:35:33.790737 29481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b991de684684727838b5b8b3e63dd23" candidate_uuid: "4e4fb9691c284161ad175ef91014eea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b24ef4df9248569569bc764209741c" is_pre_election: true
I20260923 17:35:33.790995 29343 raft_consensus.cc:2454] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4fb9691c284161ad175ef91014eea5 in term 0.
I20260923 17:35:33.791021 29481 raft_consensus.cc:2454] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4fb9691c284161ad175ef91014eea5 in term 0.
I20260923 17:35:33.791473 29137 leader_election.cc:302] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4fb9691c284161ad175ef91014eea5, 939711233b934a7692a3bd0151f4a8ec; no voters:
I20260923 17:35:33.791774 29603 raft_consensus.cc:2781] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:35:33.791879 29603 raft_consensus.cc:482] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:35:33.791975 29603 raft_consensus.cc:3037] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:33.794838 29603 raft_consensus.cc:504] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.795562 29603 leader_election.cc:288] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [CANDIDATE]: Term 1 election: Requested vote from peers 939711233b934a7692a3bd0151f4a8ec (127.20.240.66:45963), b1b24ef4df9248569569bc764209741c (127.20.240.67:46455)
I20260923 17:35:33.796087 29343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b991de684684727838b5b8b3e63dd23" candidate_uuid: "4e4fb9691c284161ad175ef91014eea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939711233b934a7692a3bd0151f4a8ec"
I20260923 17:35:33.796083 29481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b991de684684727838b5b8b3e63dd23" candidate_uuid: "4e4fb9691c284161ad175ef91014eea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b24ef4df9248569569bc764209741c"
I20260923 17:35:33.796285 29343 raft_consensus.cc:3037] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:33.796334 29481 raft_consensus.cc:3037] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:33.798344 29343 raft_consensus.cc:2454] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4fb9691c284161ad175ef91014eea5 in term 1.
I20260923 17:35:33.798789 29137 leader_election.cc:302] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4fb9691c284161ad175ef91014eea5, 939711233b934a7692a3bd0151f4a8ec; no voters:
I20260923 17:35:33.798962 29481 raft_consensus.cc:2454] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4fb9691c284161ad175ef91014eea5 in term 1.
I20260923 17:35:33.799048 29603 raft_consensus.cc:2781] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:33.799526 29603 raft_consensus.cc:686] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4fb9691c284161ad175ef91014eea5, State: Running, Role: LEADER
I20260923 17:35:33.799801 29603 consensus_queue.cc:237] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } }
I20260923 17:35:33.802604 29047 catalog_manager.cc:5495] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4fb9691c284161ad175ef91014eea5 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "4e4fb9691c284161ad175ef91014eea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4fb9691c284161ad175ef91014eea5" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 } health_report { overall_health: UNKNOWN } } }
I20260923 17:35:33.834822 29674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:33.829407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.20.240.65:34607)
0923 17:35:33.829983 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:33.829987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:33.830008 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:35:33.830114 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:33.830118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:33.830182 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:33.830260 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:35:33.830821 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:33.831723 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:33.833136 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:33.833577 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:33.833689 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:33.834105 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:35:33.834232 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:35:33.834309 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:33.834435 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 17:35:33.834504 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":304,"threads_started":1}
I20260923 17:35:33.847819 29674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:33.843014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:57221 (local address 127.20.240.65:34607)
0923 17:35:33.843293 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:33.843299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:33.843427 (+ 128us) server_negotiation.cc:407] Connection header received
0923 17:35:33.843599 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:33.843604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:33.843667 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:33.843736 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:35:33.844178 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:33.844894 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:33.846190 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:33.846561 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:33.846662 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:33.846985 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:35:33.847095 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:35:33.847273 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:33.847403 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 17:35:33.847481 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":160}
I20260923 17:35:33.847846 29677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:33.842758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34607 (local address 127.20.240.65:57221)
0923 17:35:33.843342 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:33.843357 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:33.843485 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:33.843755 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:33.843759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:33.843770 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:35:33.844042 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:33.844052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:33.845019 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:33.845024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:33.846062 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:33.846075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:33.846690 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:33.846696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:33.846828 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:33.846862 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:35:33.847292 (+ 430us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:35:33.847303 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 17:35:33.847379 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 17:35:33.847513 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":18,"thread_start_us":258,"threads_started":1}
I20260923 17:35:33.856212 29481 raft_consensus.cc:1261] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Refusing update from remote peer 4e4fb9691c284161ad175ef91014eea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:33.856338 29343 raft_consensus.cc:1261] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Refusing update from remote peer 4e4fb9691c284161ad175ef91014eea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:35:33.856756 29663 consensus_queue.cc:1035] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1b24ef4df9248569569bc764209741c" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 46455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:33.857095 29669 consensus_queue.cc:1035] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "939711233b934a7692a3bd0151f4a8ec" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:35:33.911499 29047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:33.911772 29047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:35:33.914691 29047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c18b6de8a8f54fb196e56cbc636ef6b5: Sending DeleteTablet for 3 replicas of tablet 1b991de684684727838b5b8b3e63dd23
I20260923 17:35:33.915621 29184 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b991de684684727838b5b8b3e63dd23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:33 UTC) from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:37384
I20260923 17:35:33.915694 29323 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b991de684684727838b5b8b3e63dd23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:33 UTC) from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:48184
I20260923 17:35:33.916113 29462 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b991de684684727838b5b8b3e63dd23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:35:33 UTC) from {username='oryx', principal='oryx/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:42754
I20260923 17:35:33.916448 29683 tablet_replica.cc:330] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5: stopping tablet replica
I20260923 17:35:33.916682 29683 raft_consensus.cc:2229] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:35:33.916973 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 29118
I20260923 17:35:33.917129 29684 tablet_replica.cc:330] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c: stopping tablet replica
I20260923 17:35:33.917162 29683 raft_consensus.cc:2258] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:33.917443 29684 raft_consensus.cc:2229] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:33.917775 29684 raft_consensus.cc:2258] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:33.917816 29682 tablet_replica.cc:330] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec: stopping tablet replica
I20260923 17:35:33.918053 29682 raft_consensus.cc:2229] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:35:33.918313 29682 raft_consensus.cc:2258] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:35:33.919515 29682 ts_tablet_manager.cc:1905] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:33.920046 29683 ts_tablet_manager.cc:1905] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:33.921231 29684 ts_tablet_manager.cc:1905] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:35:33.926641 29683 ts_tablet_manager.cc:1918] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:35:33.926631 29682 ts_tablet_manager.cc:1918] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:35:33.926783 29683 log.cc:1196] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/1b991de684684727838b5b8b3e63dd23
I20260923 17:35:33.926788 29682 log.cc:1196] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/1b991de684684727838b5b8b3e63dd23
I20260923 17:35:33.927318 29683 ts_tablet_manager.cc:1939] T 1b991de684684727838b5b8b3e63dd23 P 4e4fb9691c284161ad175ef91014eea5: Deleting consensus metadata
I20260923 17:35:33.927309 29682 ts_tablet_manager.cc:1939] T 1b991de684684727838b5b8b3e63dd23 P 939711233b934a7692a3bd0151f4a8ec: Deleting consensus metadata
I20260923 17:35:33.928524 29684 ts_tablet_manager.cc:1918] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:35:33.928543 29031 catalog_manager.cc:4841] TS 4e4fb9691c284161ad175ef91014eea5 (127.20.240.65:34607): tablet 1b991de684684727838b5b8b3e63dd23 (table test-table [id=aa6b3be747e746d48670b545dd17bf52]) successfully deleted
I20260923 17:35:33.928658 29684 log.cc:1196] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/1b991de684684727838b5b8b3e63dd23
I20260923 17:35:33.928818 29034 catalog_manager.cc:4841] TS 939711233b934a7692a3bd0151f4a8ec (127.20.240.66:45963): tablet 1b991de684684727838b5b8b3e63dd23 (table test-table [id=aa6b3be747e746d48670b545dd17bf52]) successfully deleted
I20260923 17:35:33.929093 29684 ts_tablet_manager.cc:1939] T 1b991de684684727838b5b8b3e63dd23 P b1b24ef4df9248569569bc764209741c: Deleting consensus metadata
Sep 23 17:35:33 dist-test-slave-ppjl krb5kdc[29002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.240.65@KRBTEST.COM
I20260923 17:35:33.930164 29033 catalog_manager.cc:4841] TS b1b24ef4df9248569569bc764209741c (127.20.240.67:46455): tablet 1b991de684684727838b5b8b3e63dd23 (table test-table [id=aa6b3be747e746d48670b545dd17bf52]) successfully deleted
I20260923 17:35:33.936705 29674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:33.918208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.20.240.65:34607)
0923 17:35:33.918355 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:33.918359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:33.918906 (+ 547us) server_negotiation.cc:407] Connection header received
0923 17:35:33.919184 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:33.919190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:33.919281 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:33.919391 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:33.921330 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:33.922444 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:33.924319 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:33.925003 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:33.925115 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:33.931263 (+ 6148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:33.931278 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:33.931295 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:33.931333 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:33.934166 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:33.934827 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:33.934834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:33.934836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:33.934993 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:33.935362 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:33.935366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:33.935368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:33.935729 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:33.935954 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:33.936201 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 17:35:33.936365 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 17:35:33.937347 29249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:34.085501 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 29118
I20260923 17:35:34.107543 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 29258
Sep 23 17:35:34 dist-test-slave-ppjl krb5kdc[29002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.240.66@KRBTEST.COM
I20260923 17:35:34.122364 29687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:34.108293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.20.240.66:45963)
0923 17:35:34.108808 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:34.108813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:34.108827 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:35:34.108877 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:34.108881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:34.108937 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:34.109027 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:34.110288 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:34.111016 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:34.112344 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:34.112751 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:34.112859 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:34.116998 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:34.117021 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:34.117037 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:34.117073 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:34.119835 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:34.120460 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:34.120469 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:34.120473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:34.120663 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:34.121020 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:34.121024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:34.121027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:34.121399 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:34.121642 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:34.121864 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 17:35:34.122037 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":235,"threads_started":1}
I20260923 17:35:34.123039 29386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:34.249128 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 29258
I20260923 17:35:34.267376 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 29397
Sep 23 17:35:34 dist-test-slave-ppjl krb5kdc[29002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.240.67@KRBTEST.COM
I20260923 17:35:34.283555 29689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:34.268141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.20.240.67:46455)
0923 17:35:34.268585 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:34.268589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:34.268606 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:35:34.268661 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:34.268664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:34.268729 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:34.268825 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:34.270098 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:34.271023 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:34.272373 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:34.272822 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:34.272930 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:34.277300 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:34.277318 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:34.277336 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:34.277385 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:34.280664 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:34.281424 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:34.281432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:34.281434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:34.281607 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:34.281986 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:34.281990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:34.281992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:34.282382 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:34.282646 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:34.282952 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 17:35:34.283144 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":221,"threads_started":1}
I20260923 17:35:34.284335 29527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:34.409204 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 29397
I20260923 17:35:34.427177 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 29015
I20260923 17:35:34.428408 29087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:34.548236 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 29015
2026-09-23T17:35:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6364 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:35:34 dist-test-slave-ppjl krb5kdc[29694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:35:34 dist-test-slave-ppjl krb5kdc[29694](info): set up 2 sockets
Sep 23 17:35:34 dist-test-slave-ppjl krb5kdc[29694](info): commencing operation
krb5kdc: starting...
W20260923 17:35:35.568580 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:35:35 dist-test-slave-ppjl krb5kdc[29694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:35:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:35:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:35:35.782187 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:34053
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:34053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:36.108542 29707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:36.108876 29707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:36.109015 29707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:36.116416 29707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:35:36.116564 29707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:36.116647 29707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:36.116729 29707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:35:36.116796 29707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:35:36.127103 29707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:34053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:34053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:36.128472 29707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:36.129812 29707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:36.143215 29713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:36.143419 29712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:36.143573 29715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:36.144565 29707 server_base.cc:1034] running on GCE node
I20260923 17:35:36.145665 29707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:36.147395 29707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:36.148634 29707 hybrid_clock.cc:648] HybridClock initialized: now 1790184936148589 us; error 49 us; skew 500 ppm
I20260923 17:35:36.149086 29707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T17:35:36Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1674 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:35:36 dist-test-slave-ppjl krb5kdc[29721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:35:36 dist-test-slave-ppjl krb5kdc[29721](info): set up 2 sockets
Sep 23 17:35:36 dist-test-slave-ppjl krb5kdc[29721](info): commencing operation
krb5kdc: starting...
W20260923 17:35:37.232663 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:35:37 dist-test-slave-ppjl krb5kdc[29721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:35:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:35:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:35:37.450591 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41869
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:41869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:37.775527 29734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:37.775839 29734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:37.775981 29734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:37.783422 29734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:35:37.783551 29734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:37.783632 29734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:37.783705 29734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:35:37.783763 29734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:35:37.794440 29734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:41869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:37.795892 29734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:37.797271 29734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:37.808851 29739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:37.808872 29740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:37.808805 29742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:37.809417 29734 server_base.cc:1034] running on GCE node
I20260923 17:35:37.810307 29734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:37.812093 29734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:37.813311 29734 hybrid_clock.cc:648] HybridClock initialized: now 1790184937813257 us; error 44 us; skew 500 ppm
I20260923 17:35:37.813673 29734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:37 dist-test-slave-ppjl krb5kdc[29721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184937, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:37.819962 29734 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:35:37.821736 29734 webserver.cc:460] Webserver started at http://127.20.240.126:34379/ using document root <none> and password file <none>
I20260923 17:35:37.822499 29734 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:37.822604 29734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:37.822875 29734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:37.826012 29734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "3e4fb92cc9974b92aee254b4a894a940"
format_stamp: "Formatted at 2026-09-23 17:35:37 on dist-test-slave-ppjl"
server_key: "7dd37ca77ca604a327ecbac3fca23136"
server_key_iv: "ffcafdd460da8674f725bd68a86a6c1a"
server_key_version: "encryptionkey@0"
I20260923 17:35:37.826746 29734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "3e4fb92cc9974b92aee254b4a894a940"
format_stamp: "Formatted at 2026-09-23 17:35:37 on dist-test-slave-ppjl"
server_key: "7dd37ca77ca604a327ecbac3fca23136"
server_key_iv: "ffcafdd460da8674f725bd68a86a6c1a"
server_key_version: "encryptionkey@0"
I20260923 17:35:37.832293 29734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:35:37.836055 29749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:37.837448 29734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:35:37.837605 29734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "3e4fb92cc9974b92aee254b4a894a940"
format_stamp: "Formatted at 2026-09-23 17:35:37 on dist-test-slave-ppjl"
server_key: "7dd37ca77ca604a327ecbac3fca23136"
server_key_iv: "ffcafdd460da8674f725bd68a86a6c1a"
server_key_version: "encryptionkey@0"
I20260923 17:35:37.837786 29734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:37.852702 29734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:37.857007 29734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:37.857249 29734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:37.875406 29734 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:41869
I20260923 17:35:37.875447 29800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:41869 every 8 connection(s)
I20260923 17:35:37.876737 29734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:35:37.880813 29801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:37.881839 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 29734
I20260923 17:35:37.882052 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:35:37.882936 21441 external_mini_cluster.cc:1428] Setting key 57f9568d568c2e890dc690e9d6881b1c
I20260923 17:35:37.895721 29801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940: Bootstrap starting.
I20260923 17:35:37.898892 29801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:37.899964 29801 log.cc:824] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:35:37 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:35:37.902943 29801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940: No bootstrap required, opened a new log
I20260923 17:35:37.910637 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:37.885152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49036 (local address 127.20.240.126:41869)
0923 17:35:37.885991 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:37.886003 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:37.886049 (+ 46us) server_negotiation.cc:407] Connection header received
0923 17:35:37.887741 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:37.887770 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:37.888178 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:37.888519 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:37.890377 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:37.891791 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:37.893535 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:37.894244 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:37.894371 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:37.902753 (+ 8382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:37.902762 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:37.902779 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:37.902825 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:37.906095 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:37.906894 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:37.906902 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:37.906909 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:37.907123 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:37.907628 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:37.907633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:37.907635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:37.908237 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:37.908541 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:37.908998 (+ 457us) server_negotiation.cc:299] Negotiation successful
0923 17:35:37.909469 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":395,"threads_started":1}
I20260923 17:35:37.911249 29801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4fb92cc9974b92aee254b4a894a940" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41869 } }
I20260923 17:35:37.911746 29801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:37.911865 29801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e4fb92cc9974b92aee254b4a894a940, State: Initialized, Role: FOLLOWER
I20260923 17:35:37.912494 29801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4fb92cc9974b92aee254b4a894a940" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41869 } }
I20260923 17:35:37.912726 29801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:35:37.912853 29801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:35:37.913007 29801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:37.915494 29801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4fb92cc9974b92aee254b4a894a940" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41869 } }
I20260923 17:35:37.915988 29801 leader_election.cc:302] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e4fb92cc9974b92aee254b4a894a940; no voters:
I20260923 17:35:37.916568 29801 leader_election.cc:288] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:35:37.916692 29806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:37.917771 29806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 1 LEADER]: Becoming Leader. State: Replica: 3e4fb92cc9974b92aee254b4a894a940, State: Running, Role: LEADER
I20260923 17:35:37.918331 29806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4fb92cc9974b92aee254b4a894a940" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41869 } }
I20260923 17:35:37.918673 29801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:35:37.921202 29807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e4fb92cc9974b92aee254b4a894a940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4fb92cc9974b92aee254b4a894a940" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41869 } } }
I20260923 17:35:37.921562 29807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:37.921612 29808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e4fb92cc9974b92aee254b4a894a940. Latest consensus state: current_term: 1 leader_uuid: "3e4fb92cc9974b92aee254b4a894a940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4fb92cc9974b92aee254b4a894a940" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41869 } } }
I20260923 17:35:37.921931 29808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:37.924150 29813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:35:37.928920 29813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:35:37.938453 29813 catalog_manager.cc:1347] Generated new cluster ID: a6c4635863d743f48ef66a3755b280ee
I20260923 17:35:37.938629 29813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:35:37.963630 29813 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:35:37.965085 29813 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:35:37.976480 29813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940: Generated new TSK 0
I20260923 17:35:37.977408 29813 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:35:38.042910 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41869
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:38.388255 29827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:38.388543 29827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:38.388657 29827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:38.396046 29827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:38.396164 29827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:38.396313 29827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:35:38.406440 29827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:38.407756 29827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:38.409068 29827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:38.422685 29832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:38.422708 29835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:38.422778 29833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:38.423502 29827 server_base.cc:1034] running on GCE node
I20260923 17:35:38.424152 29827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:38.425171 29827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:38.426383 29827 hybrid_clock.cc:648] HybridClock initialized: now 1790184938426320 us; error 53 us; skew 500 ppm
I20260923 17:35:38.426676 29827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:38 dist-test-slave-ppjl krb5kdc[29721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184938, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:38.430024 29827 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:35:38.431743 29827 webserver.cc:460] Webserver started at http://127.20.240.65:40465/ using document root <none> and password file <none>
I20260923 17:35:38.432498 29827 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:38.432607 29827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:38.432891 29827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:38.436024 29827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "ceb424a24fc0468ab249d1859a41b2d1"
format_stamp: "Formatted at 2026-09-23 17:35:38 on dist-test-slave-ppjl"
server_key: "f2a9657d25cc954e7de0863c8a73ae96"
server_key_iv: "e8ff2ff68b64e2659870aaf6a2f2ba39"
server_key_version: "encryptionkey@0"
I20260923 17:35:38.436830 29827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "ceb424a24fc0468ab249d1859a41b2d1"
format_stamp: "Formatted at 2026-09-23 17:35:38 on dist-test-slave-ppjl"
server_key: "f2a9657d25cc954e7de0863c8a73ae96"
server_key_iv: "e8ff2ff68b64e2659870aaf6a2f2ba39"
server_key_version: "encryptionkey@0"
I20260923 17:35:38.442391 29827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:35:38.446046 29842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:38.447527 29827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:35:38.447690 29827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "ceb424a24fc0468ab249d1859a41b2d1"
format_stamp: "Formatted at 2026-09-23 17:35:38 on dist-test-slave-ppjl"
server_key: "f2a9657d25cc954e7de0863c8a73ae96"
server_key_iv: "e8ff2ff68b64e2659870aaf6a2f2ba39"
server_key_version: "encryptionkey@0"
I20260923 17:35:38.447933 29827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:38.475654 29827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:38.480239 29827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:38.480530 29827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:38.481504 29827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:38.482896 29827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:38.482972 29827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:38.483086 29827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:38.483143 29827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:38.513051 29827 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:41441
I20260923 17:35:38.513139 29954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:41441 every 8 connection(s)
I20260923 17:35:38.514318 29827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:35:38.515069 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 29827
I20260923 17:35:38.515300 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:35:38.516075 21441 external_mini_cluster.cc:1428] Setting key d8834f570fe6bf6457caac16a05984bc
Sep 23 17:35:38 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184938, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:35:38.538188 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:38.517159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:52067 (local address 127.20.240.126:41869)
0923 17:35:38.517632 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:38.517639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:38.518768 (+ 1129us) server_negotiation.cc:407] Connection header received
0923 17:35:38.521378 (+ 2610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:38.521382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:38.521457 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:38.521554 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:38.523588 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:38.524440 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:38.526589 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:38.527280 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:38.527446 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:38.531599 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:38.531607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:38.531611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:38.531655 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:38.534652 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:38.535485 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:38.535491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:38.535494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:38.535641 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:38.536113 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:38.536118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:38.536121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:38.536490 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:38.536646 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:38.537569 (+ 923us) server_negotiation.cc:299] Negotiation successful
0923 17:35:38.537807 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":223,"threads_started":1}
I20260923 17:35:38.539204 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:38.517596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:52067)
0923 17:35:38.518577 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:38.518652 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:38.521007 (+ 2355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:38.521817 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:38.521828 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:38.522519 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:38.523377 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:38.523404 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:38.524614 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:38.524619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:38.526415 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:38.526429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:38.527508 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:38.527513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:38.527719 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:38.528465 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:38.528496 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:38.531344 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:38.534816 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:38.534824 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:38.534839 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:38.535357 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:38.535740 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:38.535743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:38.535745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:38.535964 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:38.536683 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:38.536691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:38.537132 (+ 441us) client_negotiation.cc:769] Sending connection context
0923 17:35:38.537407 (+ 275us) client_negotiation.cc:240] Negotiation successful
0923 17:35:38.537858 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":437,"threads_started":1}
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
I20260923 17:35:38.540879 29955 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41869
I20260923 17:35:38.541193 29955 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:38.542018 29955 heartbeater.cc:511] Master 127.20.240.126:41869 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
I20260923 17:35:38.544536 29766 ts_manager.cc:194] Registered new tserver with Master: ceb424a24fc0468ab249d1859a41b2d1 (127.20.240.65:41441)
I20260923 17:35:38.546101 29766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:52067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:35:38.570418 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41869
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:38.892333 29961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:38.892622 29961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:38.892755 29961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:38.900192 29961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:38.900306 29961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:38.900451 29961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:35:38.910642 29961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:38.912001 29961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:38.913288 29961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:38.926677 29967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:38.926683 29966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:38.926951 29969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:38.927598 29961 server_base.cc:1034] running on GCE node
I20260923 17:35:38.928267 29961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:38.929301 29961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:38.930536 29961 hybrid_clock.cc:648] HybridClock initialized: now 1790184938930484 us; error 43 us; skew 500 ppm
I20260923 17:35:38.930858 29961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:38 dist-test-slave-ppjl krb5kdc[29721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184938, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:38.934099 29961 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:35:38.935577 29961 webserver.cc:460] Webserver started at http://127.20.240.66:43713/ using document root <none> and password file <none>
I20260923 17:35:38.936282 29961 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:38.936400 29961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:38.936694 29961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:38.939765 29961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "c250defef96e4cb2bea2bb6072cdac12"
format_stamp: "Formatted at 2026-09-23 17:35:38 on dist-test-slave-ppjl"
server_key: "e2576ca38f4ecb9dfd6da72ae08dd923"
server_key_iv: "0d514c3906fae58348281b34c6d9d0b9"
server_key_version: "encryptionkey@0"
I20260923 17:35:38.940585 29961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "c250defef96e4cb2bea2bb6072cdac12"
format_stamp: "Formatted at 2026-09-23 17:35:38 on dist-test-slave-ppjl"
server_key: "e2576ca38f4ecb9dfd6da72ae08dd923"
server_key_iv: "0d514c3906fae58348281b34c6d9d0b9"
server_key_version: "encryptionkey@0"
I20260923 17:35:38.946049 29961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:35:38.949726 29976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:38.951294 29961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:35:38.951476 29961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "c250defef96e4cb2bea2bb6072cdac12"
format_stamp: "Formatted at 2026-09-23 17:35:38 on dist-test-slave-ppjl"
server_key: "e2576ca38f4ecb9dfd6da72ae08dd923"
server_key_iv: "0d514c3906fae58348281b34c6d9d0b9"
server_key_version: "encryptionkey@0"
I20260923 17:35:38.951674 29961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:38.987473 29961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:38.991808 29961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:38.992069 29961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:38.993043 29961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:38.994416 29961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:38.994489 29961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:38.994580 29961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:38.994622 29961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:39.026060 29961 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:37549
I20260923 17:35:39.026118 30088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:37549 every 8 connection(s)
I20260923 17:35:39.027472 29961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:35:39.031476 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 29961
I20260923 17:35:39.031651 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:35:39.032133 21441 external_mini_cluster.cc:1428] Setting key c87d4689a564e1b7d7478d00caa7f309
Sep 23 17:35:39 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184938, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:35:39.050736 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:39.030022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:47635 (local address 127.20.240.126:41869)
0923 17:35:39.030228 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:39.030233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:39.031543 (+ 1310us) server_negotiation.cc:407] Connection header received
0923 17:35:39.033598 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:39.033604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:39.033689 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:39.033779 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:39.035951 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:39.036669 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:39.038930 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:39.039494 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:39.039604 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:39.044120 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:39.044126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:39.044131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:39.044177 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:39.047332 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:39.048246 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:39.048251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:39.048254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:39.048413 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:39.048827 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:39.048832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:39.048835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:39.049183 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:39.049333 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:39.050119 (+ 786us) server_negotiation.cc:299] Negotiation successful
0923 17:35:39.050367 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:35:39.051921 30091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:39.030621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:47635)
0923 17:35:39.031390 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:39.031429 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:39.033342 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:39.034079 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:39.034091 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:39.034826 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:39.035669 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:39.035697 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:39.036823 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:39.036828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:39.038660 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:39.038678 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:39.039749 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:39.039756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:39.040089 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:39.040874 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:39.040911 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:39.043794 (+ 2883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:39.047509 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:39.047517 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:39.047534 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:39.048099 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:39.048517 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:39.048520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:39.048523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:39.048717 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:39.049313 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:39.049320 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:39.049681 (+ 361us) client_negotiation.cc:769] Sending connection context
0923 17:35:39.049963 (+ 282us) client_negotiation.cc:240] Negotiation successful
0923 17:35:39.050447 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":327,"threads_started":1}
I20260923 17:35:39.053516 30089 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41869
I20260923 17:35:39.053839 30089 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:39.054625 30089 heartbeater.cc:511] Master 127.20.240.126:41869 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
I20260923 17:35:39.056392 29766 ts_manager.cc:194] Registered new tserver with Master: c250defef96e4cb2bea2bb6072cdac12 (127.20.240.66:37549)
I20260923 17:35:39.057390 29766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:47635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:35:39.086495 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41869
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:39.429221 30094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:39.429524 30094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:39.429646 30094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:39.437071 30094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:39.437191 30094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:39.437340 30094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:35:39.447149 30094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:39.448587 30094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:39.449843 30094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:39.463047 30099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:39.462972 30102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:39.463006 30100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:39.463693 30094 server_base.cc:1034] running on GCE node
I20260923 17:35:39.464301 30094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:39.465220 30094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:39.466408 30094 hybrid_clock.cc:648] HybridClock initialized: now 1790184939466364 us; error 39 us; skew 500 ppm
I20260923 17:35:39.466717 30094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:39 dist-test-slave-ppjl krb5kdc[29721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184939, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:39.469681 30094 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:35:39.471231 30094 webserver.cc:460] Webserver started at http://127.20.240.67:42729/ using document root <none> and password file <none>
I20260923 17:35:39.471884 30094 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:39.471980 30094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:39.472234 30094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:39.475011 30094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "7d1e671dd80d4381b41df0ce17960968"
format_stamp: "Formatted at 2026-09-23 17:35:39 on dist-test-slave-ppjl"
server_key: "6a0f76d10635e1e44eeb9d9d6372dbc8"
server_key_iv: "50433892b152bd1d953488d3f06acdbf"
server_key_version: "encryptionkey@0"
I20260923 17:35:39.475802 30094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "7d1e671dd80d4381b41df0ce17960968"
format_stamp: "Formatted at 2026-09-23 17:35:39 on dist-test-slave-ppjl"
server_key: "6a0f76d10635e1e44eeb9d9d6372dbc8"
server_key_iv: "50433892b152bd1d953488d3f06acdbf"
server_key_version: "encryptionkey@0"
I20260923 17:35:39.480836 30094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:35:39.484328 30109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:39.485714 30094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:35:39.485867 30094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "7d1e671dd80d4381b41df0ce17960968"
format_stamp: "Formatted at 2026-09-23 17:35:39 on dist-test-slave-ppjl"
server_key: "6a0f76d10635e1e44eeb9d9d6372dbc8"
server_key_iv: "50433892b152bd1d953488d3f06acdbf"
server_key_version: "encryptionkey@0"
I20260923 17:35:39.486052 30094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:39.524588 30094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:39.528707 30094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:39.528983 30094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:39.530205 30094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:39.531699 30094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:39.531770 30094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:39.531884 30094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:39.531934 30094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:39.549826 29955 heartbeater.cc:503] Master 127.20.240.126:41869 was elected leader, sending a full tablet report...
I20260923 17:35:39.563825 30094 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:45625
I20260923 17:35:39.563861 30221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:45625 every 8 connection(s)
I20260923 17:35:39.565102 30094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:35:39.566953 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 30094
I20260923 17:35:39.567135 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:35:39.567622 21441 external_mini_cluster.cc:1428] Setting key 40255cfb2c1fcbce64c1b7b74958f1e2
Sep 23 17:35:39 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184939, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:35:39.585722 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:39.567835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:38991 (local address 127.20.240.126:41869)
0923 17:35:39.568310 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:39.568316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:39.569399 (+ 1083us) server_negotiation.cc:407] Connection header received
0923 17:35:39.571580 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:39.571584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:39.571656 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:39.571761 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:39.573272 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:39.574119 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:39.575823 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:39.576224 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:39.576326 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:39.579810 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:39.579815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:39.579819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:39.579926 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:39.582781 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:39.583641 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:39.583645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:39.583648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:39.583794 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:39.584187 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:39.584191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:39.584193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:39.584476 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:39.584592 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:39.585252 (+ 660us) server_negotiation.cc:299] Negotiation successful
0923 17:35:39.585434 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":224,"threads_started":1}
I20260923 17:35:39.586625 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:39.568470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:38991)
0923 17:35:39.569233 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:39.569269 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:39.571314 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:39.571986 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:39.571995 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:39.572536 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:39.573103 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:39.573120 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:39.574276 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:39.574280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:39.575692 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:39.575705 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:39.576457 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:39.576463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:39.576674 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:39.577314 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:39.577340 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:39.579584 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:39.582924 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:39.582933 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:39.582962 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:39.583503 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:39.583880 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:35:39.583883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:35:39.583886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:35:39.584082 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:35:39.584571 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:39.584578 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:39.584900 (+ 322us) client_negotiation.cc:769] Sending connection context
0923 17:35:39.585135 (+ 235us) client_negotiation.cc:240] Negotiation successful
0923 17:35:39.585531 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":285,"threads_started":1}
I20260923 17:35:39.587971 30222 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41869
I20260923 17:35:39.588267 30222 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:39.589005 30222 heartbeater.cc:511] Master 127.20.240.126:41869 requested a full tablet report, sending...
I20260923 17:35:39.590415 29766 ts_manager.cc:194] Registered new tserver with Master: 7d1e671dd80d4381b41df0ce17960968 (127.20.240.67:45625)
I20260923 17:35:39.591290 29766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:38991
I20260923 17:35:39.595547 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.65.keytab.
Entry for principal oryx/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.65.keytab.
WARNING: no policy specified for oryx/127.20.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.66.keytab.
Entry for principal oryx/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.66.keytab.
WARNING: no policy specified for oryx/127.20.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.67.keytab.
Entry for principal oryx/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.67.keytab.
I20260923 17:35:39.730989 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 29827
Sep 23 17:35:39 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:35:39.748880 30233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:39.732372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53968 (local address 127.20.240.65:41441)
0923 17:35:39.733059 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:39.733066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:39.733094 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:35:39.733155 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:39.733162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:39.733364 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:39.733521 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:39.735032 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:39.736289 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:39.737845 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:39.738446 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:39.738552 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:39.742984 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:39.743003 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:39.743019 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:39.743073 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:39.746138 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:39.746870 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:39.746878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:39.746880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:39.747088 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:39.747469 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:39.747472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:39.747474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:39.747843 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:39.748137 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:39.748382 (+ 245us) server_negotiation.cc:299] Negotiation successful
0923 17:35:39.748591 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":370,"threads_started":1}
I20260923 17:35:39.750272 29950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:39.833863 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 29827
I20260923 17:35:39.848006 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 29961
Sep 23 17:35:39 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:35:39.865383 30235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:39.848887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46118 (local address 127.20.240.66:37549)
0923 17:35:39.849392 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:39.849399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:39.849427 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:35:39.849494 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:39.849502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:39.849719 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:39.849857 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:39.851126 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:39.852192 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:39.853492 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:39.854069 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:39.854169 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:39.858658 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:39.858672 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:39.858687 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:39.858727 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:39.862167 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:39.862948 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:39.862956 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:39.862958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:39.863120 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:39.863736 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:39.863741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:39.863743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:39.864164 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:39.864446 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:39.864729 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 17:35:39.864966 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":239,"threads_started":1}
I20260923 17:35:39.866636 30084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:39.949131 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 29961
I20260923 17:35:39.962093 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 30094
Sep 23 17:35:39 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:35:39.979511 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:39.963049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.20.240.67:45625)
0923 17:35:39.963728 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:39.963735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:39.963767 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:35:39.963833 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:39.963840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:39.964066 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:39.964200 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:39.965558 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:39.966666 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:39.968211 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:39.968789 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:39.968895 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:39.973523 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:39.973537 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:39.973554 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:39.973599 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:39.976580 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:39.977366 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:39.977374 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:39.977377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:39.977521 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:39.977970 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:39.977973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:39.977975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:39.978319 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:39.978560 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:39.978832 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 17:35:39.979140 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":343,"threads_started":1}
I20260923 17:35:39.981001 30217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:40.065169 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 30094
I20260923 17:35:40.077925 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 29734
I20260923 17:35:40.079144 29796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:40.188760 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 29734
I20260923 17:35:40.205982 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41869
--webserver_interface=127.20.240.126
--webserver_port=34379
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:41869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:35:40.526991 30240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:40.527361 30240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:40.527504 30240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:40.534772 30240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:35:40.534904 30240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:40.534977 30240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:40.535053 30240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:35:40.535112 30240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:35:40.545321 30240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:41869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=34379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:40.546911 30240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:40.548208 30240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:40.560187 30245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:40.560214 30246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:40.560364 30248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:40.561074 30240 server_base.cc:1034] running on GCE node
I20260923 17:35:40.561837 30240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:40.563287 30240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:40.564508 30240 hybrid_clock.cc:648] HybridClock initialized: now 1790184940564459 us; error 37 us; skew 500 ppm
I20260923 17:35:40.564895 30240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:40 dist-test-slave-ppjl krb5kdc[29721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184940, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:40.568027 30240 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:35:40.569505 30240 webserver.cc:460] Webserver started at http://127.20.240.126:34379/ using document root <none> and password file <none>
I20260923 17:35:40.570201 30240 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:40.570348 30240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:40.576315 30240 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:35:40.579501 30255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:40.581216 30240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:35:40.581420 30240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "3e4fb92cc9974b92aee254b4a894a940"
format_stamp: "Formatted at 2026-09-23 17:35:37 on dist-test-slave-ppjl"
server_key: "7dd37ca77ca604a327ecbac3fca23136"
server_key_iv: "ffcafdd460da8674f725bd68a86a6c1a"
server_key_version: "encryptionkey@0"
I20260923 17:35:40.582479 30240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:40.596292 30240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:40.600809 30240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:40.601082 30240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:40.618719 30240 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:41869
I20260923 17:35:40.618737 30306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:41869 every 8 connection(s)
I20260923 17:35:40.620183 30240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:35:40.627271 30307 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:35:40.627442 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 30240
I20260923 17:35:40.628391 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:41441
--local_ip_for_outbound_sockets=127.20.240.65
--tserver_master_addrs=127.20.240.126:41869
--webserver_port=40465
--webserver_interface=127.20.240.65
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.240.65
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:35:40.631878 30307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940: Bootstrap starting.
I20260923 17:35:40.652055 30307 log.cc:824] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:40.661234 30307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:35:40.661759 30307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940: Bootstrap complete.
I20260923 17:35:40.670791 30307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4fb92cc9974b92aee254b4a894a940" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41869 } }
I20260923 17:35:40.671751 30307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e4fb92cc9974b92aee254b4a894a940, State: Initialized, Role: FOLLOWER
I20260923 17:35:40.672413 30307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4fb92cc9974b92aee254b4a894a940" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41869 } }
I20260923 17:35:40.672639 30307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:35:40.672763 30307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:35:40.672926 30307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:35:40.677006 30307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4fb92cc9974b92aee254b4a894a940" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41869 } }
I20260923 17:35:40.677541 30307 leader_election.cc:302] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e4fb92cc9974b92aee254b4a894a940; no voters:
I20260923 17:35:40.678148 30307 leader_election.cc:288] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:35:40.678391 30311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:35:40.679520 30311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [term 2 LEADER]: Becoming Leader. State: Replica: 3e4fb92cc9974b92aee254b4a894a940, State: Running, Role: LEADER
I20260923 17:35:40.680357 30311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4fb92cc9974b92aee254b4a894a940" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41869 } }
I20260923 17:35:40.680637 30307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:35:40.684456 30313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e4fb92cc9974b92aee254b4a894a940. Latest consensus state: current_term: 2 leader_uuid: "3e4fb92cc9974b92aee254b4a894a940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4fb92cc9974b92aee254b4a894a940" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41869 } } }
I20260923 17:35:40.684827 30313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:40.685310 30312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3e4fb92cc9974b92aee254b4a894a940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4fb92cc9974b92aee254b4a894a940" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41869 } } }
I20260923 17:35:40.685580 30312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940 [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:40.696225 30321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:35:40.702013 30321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:35:40.703478 30321 catalog_manager.cc:1259] Loaded cluster ID: a6c4635863d743f48ef66a3755b280ee
I20260923 17:35:40.703588 30321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:35:40.707598 30321 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:35:40.709128 30321 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3e4fb92cc9974b92aee254b4a894a940: Loaded TSK: 0
I20260923 17:35:40.710666 30321 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:35:40.970721 30309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:40.971041 30309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:40.971192 30309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:40.978595 30309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:40.978711 30309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:40.978850 30309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:35:40.989617 30309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.65.keytab
--principal=oryx/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:41441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=40465
--tserver_master_addrs=127.20.240.126:41869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:40.991056 30309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:40.992401 30309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:41.006860 30334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:41.006834 30335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:41.007120 30337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:41.008407 30309 server_base.cc:1034] running on GCE node
I20260923 17:35:41.009125 30309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:41.010325 30309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:41.011559 30309 hybrid_clock.cc:648] HybridClock initialized: now 1790184941011495 us; error 54 us; skew 500 ppm
I20260923 17:35:41.011937 30309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184941, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:41.015885 30309 init.cc:377] Logged in from keytab as oryx/127.20.240.65@KRBTEST.COM (short username oryx)
I20260923 17:35:41.017498 30309 webserver.cc:460] Webserver started at http://127.20.240.65:40465/ using document root <none> and password file <none>
I20260923 17:35:41.018173 30309 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:41.018303 30309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:41.024155 30309 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:35:41.027325 30344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:41.028918 30309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:35:41.029145 30309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "ceb424a24fc0468ab249d1859a41b2d1"
format_stamp: "Formatted at 2026-09-23 17:35:38 on dist-test-slave-ppjl"
server_key: "f2a9657d25cc954e7de0863c8a73ae96"
server_key_iv: "e8ff2ff68b64e2659870aaf6a2f2ba39"
server_key_version: "encryptionkey@0"
I20260923 17:35:41.030136 30309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:41.052287 30309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:41.057046 30309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:41.057331 30309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:41.058395 30309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:41.059844 30309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:41.059962 30309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:41.060075 30309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:41.060144 30309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:41.090893 30309 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:41441
I20260923 17:35:41.090963 30456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:41441 every 8 connection(s)
I20260923 17:35:41.092423 30309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:35:41.092998 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 30309
I20260923 17:35:41.093845 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:37549
--local_ip_for_outbound_sockets=127.20.240.66
--tserver_master_addrs=127.20.240.126:41869
--webserver_port=43713
--webserver_interface=127.20.240.66
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.240.66
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:41.112485 30460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:41.095648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:45315)
0923 17:35:41.096716 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:41.096762 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:41.098945 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:41.100510 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:41.100521 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:41.101168 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:41.101716 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:41.101734 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.103987 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.103992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:41.105294 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:41.105307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.106653 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.106658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:41.106856 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:41.107589 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:41.107618 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:41.111484 (+ 3866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":589,"threads_started":1}
W20260923 17:35:41.113230 30457 heartbeater.cc:644] Failed to heartbeat to 127.20.240.126:41869 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.240.126:41869: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
W20260923 17:35:41.113312 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:41.095646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:45315 (local address 127.20.240.126:41869)
0923 17:35:41.096406 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:41.096422 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:41.096903 (+ 481us) server_negotiation.cc:407] Connection header received
0923 17:35:41.099578 (+ 2675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:41.099594 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:41.099965 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:41.100297 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:41.101926 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.103806 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.105435 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.106415 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.106599 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:41.111960 (+ 5361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:45315: BlockingRecv error: recv got EOF from 127.20.240.65:45315 (error 108)
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":253,"threads_started":1}
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:41.122632 30460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:41.113969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:38411)
0923 17:35:41.114166 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:41.114182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:41.114273 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:41.114576 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:41.114579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:41.114891 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:41.115037 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:41.115046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.116504 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.116509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:41.117536 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:41.117551 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.118514 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.118519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:41.118659 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:41.119328 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:41.119347 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:41.122414 (+ 3067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 17:35:41.122942 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:41.114047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:38411 (local address 127.20.240.126:41869)
0923 17:35:41.114265 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:41.114270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:41.114292 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:35:41.114361 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:41.114366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:41.114446 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:41.114554 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:41.115208 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.116314 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.117695 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.118336 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.118457 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:41.122705 (+ 4248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:38411: BlockingRecv error: recv got EOF from 127.20.240.65:38411 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:41.131372 30460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:41.123391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:41533)
0923 17:35:41.123575 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:41.123596 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:41.123703 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:41.124022 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:41.124027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:41.124343 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:41.124503 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:41.124512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.125516 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.125521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:41.126471 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:41.126486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.127223 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.127230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:41.127415 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:41.127992 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:41.128009 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:41.131026 (+ 3017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":29}
W20260923 17:35:41.131557 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:41.123544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:41533 (local address 127.20.240.126:41869)
0923 17:35:41.123698 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:41.123704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:41.123722 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:35:41.123806 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:41.123811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:41.123890 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:41.123978 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:41.124640 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.125375 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.126599 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.127034 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.127145 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:41.131275 (+ 4130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:41533: BlockingRecv error: recv got EOF from 127.20.240.65:41533 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260923 17:35:41.131752 30457 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:35:41.421175 30461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:41.421485 30461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:41.421602 30461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:41.429081 30461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:41.429201 30461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:41.429353 30461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:35:41.439596 30461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.66.keytab
--principal=oryx/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:37549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=43713
--tserver_master_addrs=127.20.240.126:41869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:41.440991 30461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:41.442312 30461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:41.455439 30466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:41.455518 30467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:41.455484 30469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:41.456586 30461 server_base.cc:1034] running on GCE node
I20260923 17:35:41.457209 30461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:41.458173 30461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:41.459369 30461 hybrid_clock.cc:648] HybridClock initialized: now 1790184941459310 us; error 48 us; skew 500 ppm
I20260923 17:35:41.459713 30461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184941, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:41.462868 30461 init.cc:377] Logged in from keytab as oryx/127.20.240.66@KRBTEST.COM (short username oryx)
I20260923 17:35:41.464454 30461 webserver.cc:460] Webserver started at http://127.20.240.66:43713/ using document root <none> and password file <none>
I20260923 17:35:41.465137 30461 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:41.465257 30461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:41.470657 30461 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:35:41.473474 30476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:41.474946 30461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:35:41.475101 30461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "c250defef96e4cb2bea2bb6072cdac12"
format_stamp: "Formatted at 2026-09-23 17:35:38 on dist-test-slave-ppjl"
server_key: "e2576ca38f4ecb9dfd6da72ae08dd923"
server_key_iv: "0d514c3906fae58348281b34c6d9d0b9"
server_key_version: "encryptionkey@0"
I20260923 17:35:41.476092 30461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:41.493829 30461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:41.497813 30461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:41.498059 30461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:41.499044 30461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:41.500460 30461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:41.500578 30461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:41.500685 30461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:41.500752 30461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:41.532099 30461 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:37549
I20260923 17:35:41.532193 30588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:37549 every 8 connection(s)
I20260923 17:35:41.533631 30461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:35:41.538396 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 30461
I20260923 17:35:41.539189 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:45625
--local_ip_for_outbound_sockets=127.20.240.67
--tserver_master_addrs=127.20.240.126:41869
--webserver_port=42729
--webserver_interface=127.20.240.67
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.240.67
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:35:41.552007 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:41.536270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:44633 (local address 127.20.240.126:41869)
0923 17:35:41.536480 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:41.536485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:41.537775 (+ 1290us) server_negotiation.cc:407] Connection header received
0923 17:35:41.540360 (+ 2585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:41.540364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:41.540440 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:41.540551 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:41.542635 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.543498 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.545755 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.546192 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.546301 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:41.551757 (+ 5456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:44633: BlockingRecv error: recv got EOF from 127.20.240.66:44633 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:35:41.552598 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:41.536835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:44633)
0923 17:35:41.537598 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:41.537636 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:41.540019 (+ 2383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:41.540838 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:41.540850 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:41.541580 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:41.542403 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:41.542430 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.543694 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.543699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:41.545553 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:41.545571 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.546444 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.546450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:41.546677 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:41.547458 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:41.547484 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:41.551483 (+ 3999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":298,"threads_started":1}
W20260923 17:35:41.553396 30589 heartbeater.cc:644] Failed to heartbeat to 127.20.240.126:41869 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.240.126:41869: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:41.562031 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:41.554075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:39165)
0923 17:35:41.554254 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:41.554269 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:41.554387 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:41.554714 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:41.554719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:41.555081 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:41.555264 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:41.555274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.556346 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.556350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:41.557255 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:41.557270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.557983 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.557988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:41.558142 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:41.558791 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:41.558822 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:41.561807 (+ 2985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 17:35:41.562664 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:41.554257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:39165 (local address 127.20.240.126:41869)
0923 17:35:41.554444 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:41.554450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:41.554469 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:35:41.554527 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:41.554530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:41.554589 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:41.554667 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:41.555440 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.556204 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.557404 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.557825 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.557933 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:41.562096 (+ 4163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:39165: BlockingRecv error: recv got EOF from 127.20.240.66:39165 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:41.571512 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:41.562689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:58357)
0923 17:35:41.562925 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:41.562943 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:41.563042 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:41.563561 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:41.563564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:41.563840 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:41.564017 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:41.564027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.565398 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.565401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:41.566315 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:41.566338 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.567276 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.567283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:41.567471 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:41.568054 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:41.568072 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:41.571243 (+ 3171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133}
W20260923 17:35:41.571727 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:41.562799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:58357 (local address 127.20.240.126:41869)
0923 17:35:41.562953 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:41.562959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:41.562981 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:35:41.563229 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:41.563236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:41.563315 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:41.563408 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:41.564152 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.565245 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.566455 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.567096 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.567277 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:41.571485 (+ 4208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:58357: BlockingRecv error: recv got EOF from 127.20.240.66:58357 (error 108)
Metrics: {"server-negotiator.queue_time_us":50,"thread_start_us":343,"threads_started":1}
W20260923 17:35:41.571870 30589 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:35:41.866712 30592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:41.867022 30592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:41.867151 30592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:41.874614 30592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:41.874738 30592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:41.874902 30592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:35:41.885576 30592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/krb5kdc/oryx_127.20.240.67.keytab
--principal=oryx/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:45625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=42729
--tserver_master_addrs=127.20.240.126:41869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:41.886991 30592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:41.888285 30592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:41.901201 30598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:41.901291 30599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:41.901328 30601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:41.902245 30592 server_base.cc:1034] running on GCE node
I20260923 17:35:41.903002 30592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:41.904003 30592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:41.905220 30592 hybrid_clock.cc:648] HybridClock initialized: now 1790184941905160 us; error 45 us; skew 500 ppm
I20260923 17:35:41.905530 30592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184941, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:35:41.908795 30592 init.cc:377] Logged in from keytab as oryx/127.20.240.67@KRBTEST.COM (short username oryx)
I20260923 17:35:41.910326 30592 webserver.cc:460] Webserver started at http://127.20.240.67:42729/ using document root <none> and password file <none>
I20260923 17:35:41.911031 30592 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:41.911310 30592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:41.916882 30592 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:35:41.919624 30608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:41.921036 30592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:35:41.921192 30592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "7d1e671dd80d4381b41df0ce17960968"
format_stamp: "Formatted at 2026-09-23 17:35:39 on dist-test-slave-ppjl"
server_key: "6a0f76d10635e1e44eeb9d9d6372dbc8"
server_key_iv: "50433892b152bd1d953488d3f06acdbf"
server_key_version: "encryptionkey@0"
I20260923 17:35:41.922188 30592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:41.946404 30592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:41.950706 30592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:41.951000 30592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:41.952095 30592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:41.953564 30592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:41.953670 30592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:41.953778 30592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:41.953845 30592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:41.984664 30592 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:45625
I20260923 17:35:41.984732 30720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:45625 every 8 connection(s)
I20260923 17:35:41.986068 30592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:35:41.994215 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 30592
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:41 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:35:42.003259 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:41.988629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:54749 (local address 127.20.240.126:41869)
0923 17:35:41.988840 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:41.988845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:41.990087 (+ 1242us) server_negotiation.cc:407] Connection header received
0923 17:35:41.992029 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:41.992035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:41.992125 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:41.992215 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:41.993774 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.994540 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.996591 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.996992 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.997083 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:42.003025 (+ 5942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:54749: BlockingRecv error: recv got EOF from 127.20.240.67:54749 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:35:42.004341 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:41.989121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:54749)
0923 17:35:41.989822 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:41.989935 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:41.991738 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:41.992482 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:41.992491 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:41.993022 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:41.993598 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:41.993617 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.994686 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.994690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:41.996468 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:41.996484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.997222 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:41.997230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:41.997538 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:41.998331 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:41.998362 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:42.002815 (+ 4453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":271,"threads_started":1}
W20260923 17:35:42.005134 30721 heartbeater.cc:644] Failed to heartbeat to 127.20.240.126:41869 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.240.126:41869: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
I20260923 17:35:42.011946 30593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:41.995326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.20.240.126:41869)
0923 17:35:41.995539 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:41.995545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:41.995948 (+ 403us) server_negotiation.cc:407] Connection header received
0923 17:35:41.996161 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:41.996166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:41.996242 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:41.996336 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:41.998097 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:41.998916 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.000651 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.001261 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.001358 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:42.004637 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:42.004646 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:42.004664 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:42.004705 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:42.008532 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:42.009215 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:42.009223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:42.009230 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:42.009476 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:35:42.009967 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:35:42.009972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:35:42.009975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:35:42.010489 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:42.010777 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:42.011295 (+ 518us) server_negotiation.cc:299] Negotiation successful
0923 17:35:42.011674 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
Sep 23 17:35:42 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:42 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:42.014295 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:42.005771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:39561)
0923 17:35:42.005949 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:42.005975 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:42.006059 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:42.006395 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:42.006399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:42.006785 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:42.006931 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:42.006941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.008312 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.008316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:42.009206 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:42.009218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.010022 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.010028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:42.010183 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:42.010762 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:42.010782 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:42.013974 (+ 3192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 17:35:42.014410 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:42.005860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:39561 (local address 127.20.240.126:41869)
0923 17:35:42.006044 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:42.006049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:42.006067 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:35:42.006180 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:42.006186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:42.006259 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:42.006372 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:42.007068 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.008176 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.009326 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.009852 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.009957 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:42.014169 (+ 4212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:39561: BlockingRecv error: recv got EOF from 127.20.240.67:39561 (error 108)
Metrics: {"server-negotiator.queue_time_us":68,"spinlock_wait_cycles":2432}
Sep 23 17:35:42 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:42 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:42.023559 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:42.015210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:52333)
0923 17:35:42.015445 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:42.015466 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:42.015584 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:42.015896 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:42.015899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:42.016182 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:42.016354 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:42.016363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.017393 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.017398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:42.018424 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:42.018438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.019299 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.019306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:42.019455 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:42.020106 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:42.020126 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:42.023213 (+ 3087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 17:35:42.023952 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:42.015295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:52333 (local address 127.20.240.126:41869)
0923 17:35:42.015553 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:42.015558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:42.015577 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:35:42.015692 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:42.015698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:42.015772 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:42.015845 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:42.016488 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.017211 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.018576 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.019080 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.019232 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:42.023481 (+ 4249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:52333: BlockingRecv error: recv got EOF from 127.20.240.67:52333 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
W20260923 17:35:42.023998 30721 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 17:35:42 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:42 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:42.144048 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:42.132383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:58765)
0923 17:35:42.133053 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:42.133071 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:42.133221 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:42.133600 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:42.133605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:42.134138 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:42.134362 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:42.134375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.135513 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.135520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:42.137020 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:42.137038 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.138083 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.138093 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:42.138320 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:42.139140 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:42.139193 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:42.143652 (+ 4459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":305,"threads_started":1}
W20260923 17:35:42.144730 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:42.132540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:58765 (local address 127.20.240.126:41869)
0923 17:35:42.132782 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:42.132790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:42.133165 (+ 375us) server_negotiation.cc:407] Connection header received
0923 17:35:42.133353 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:42.133357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:42.133461 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:42.133562 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:42.134529 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.135362 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.137192 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.137818 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.137956 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:42.144498 (+ 6542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:58765: BlockingRecv error: recv got EOF from 127.20.240.65:58765 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 23 17:35:42 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:42 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:42.580410 30726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:42.572428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:44239)
0923 17:35:42.572914 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:42.572928 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:42.573051 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:42.573329 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:42.573332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:42.573798 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:42.573990 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:42.574001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.574988 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.574992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:42.575983 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:42.575996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.576708 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.576713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:42.576836 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:42.577353 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:42.577369 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:42.580140 (+ 2771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":257,"threads_started":1}
W20260923 17:35:42.580658 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:42.572534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:44239 (local address 127.20.240.126:41869)
0923 17:35:42.572691 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:42.572695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:42.573007 (+ 312us) server_negotiation.cc:407] Connection header received
0923 17:35:42.573144 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:42.573148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:42.573219 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:42.573301 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:42.574116 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.574844 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.576119 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:42.576545 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:42.576637 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:42.580425 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:44239: BlockingRecv error: recv got EOF from 127.20.240.66:44239 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 23 17:35:43 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:43 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:43.038226 30727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:43.024725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:53239)
0923 17:35:43.025247 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:43.025266 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:43.025420 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:43.025804 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:43.025810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:43.026502 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:43.026705 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:43.026716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:43.027927 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:43.027932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:43.028933 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:43.028947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:43.029686 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:43.029691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:43.029822 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:43.034734 (+ 4912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:43.034755 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:43.037918 (+ 3163us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":274,"threads_started":1}
W20260923 17:35:43.038407 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:43.024926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:53239 (local address 127.20.240.126:41869)
0923 17:35:43.025118 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:43.025124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:43.025356 (+ 232us) server_negotiation.cc:407] Connection header received
0923 17:35:43.025548 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:43.025553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:43.025639 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:43.025735 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:43.026855 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:43.027762 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:43.029073 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:43.029505 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:43.029599 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:43.038174 (+ 8575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:53239: BlockingRecv error: recv got EOF from 127.20.240.67:53239 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 17:35:43 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:43 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:43.154090 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:43.145366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:49825)
0923 17:35:43.145934 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:43.145950 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:43.146100 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:43.146463 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:43.146468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:43.146967 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:43.147211 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:43.147225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:43.148528 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:43.148533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:43.149686 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:43.149700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:43.150405 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:43.150410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:43.150524 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:43.151112 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:43.151129 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:43.153843 (+ 2714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":302,"threads_started":1}
W20260923 17:35:43.154323 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:43.145493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:49825 (local address 127.20.240.126:41869)
0923 17:35:43.145738 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:43.145744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:43.146031 (+ 287us) server_negotiation.cc:407] Connection header received
0923 17:35:43.146220 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:43.146226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:43.146306 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:43.146412 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:43.147398 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:43.148370 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:43.149815 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:43.150264 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:43.150358 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:43.154080 (+ 3722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:49825: BlockingRecv error: recv got EOF from 127.20.240.65:49825 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 23 17:35:43 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:43 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:43.591137 30729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:43.581693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:56395)
0923 17:35:43.582368 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:43.582391 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:43.582564 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:43.583081 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:43.583087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:43.583681 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:43.583998 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:43.584014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:43.585265 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:43.585271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:43.586338 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:43.586373 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:43.587241 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:43.587247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:43.587380 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:43.588102 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:43.588121 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:43.590883 (+ 2762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":359,"threads_started":1}
W20260923 17:35:43.591603 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:43.581851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:56395 (local address 127.20.240.126:41869)
0923 17:35:43.582060 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:43.582067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:43.582516 (+ 449us) server_negotiation.cc:407] Connection header received
0923 17:35:43.582695 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:43.582699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:43.582797 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:43.583016 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:43.584186 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:43.585114 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:43.586518 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:43.587061 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:43.587191 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:43.591149 (+ 3958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:56395: BlockingRecv error: recv got EOF from 127.20.240.66:56395 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 23 17:35:44 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:44 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:44.049369 30730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:44.039460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:46851)
0923 17:35:44.040043 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:44.040060 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:44.040183 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:44.040652 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:44.040657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:44.041101 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:44.041285 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:44.041295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:44.042524 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:44.042530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:44.043726 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:44.043744 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:44.044762 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:44.044767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:44.044974 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:44.045628 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:44.045648 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:44.049047 (+ 3399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":283,"threads_started":1}
W20260923 17:35:44.049549 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:44.039603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:46851 (local address 127.20.240.126:41869)
0923 17:35:44.039833 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:44.039839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:44.040137 (+ 298us) server_negotiation.cc:407] Connection header received
0923 17:35:44.040333 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:44.040338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:44.040428 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:44.040547 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:44.041524 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:44.042335 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:44.044042 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:44.044581 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:44.044700 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:44.049279 (+ 4579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:46851: BlockingRecv error: recv got EOF from 127.20.240.67:46851 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 23 17:35:44 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:44 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:44.163550 30731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:44.155085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:42563)
0923 17:35:44.155628 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:44.155643 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:44.155804 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:44.156145 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:44.156149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:44.156490 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:44.156658 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:44.156667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:44.157627 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:44.157631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:44.158548 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:44.158560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:44.159346 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:44.159351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:44.159489 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:44.160107 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:44.160125 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:44.163289 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":302,"threads_started":1}
W20260923 17:35:44.163699 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:44.155172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:42563 (local address 127.20.240.126:41869)
0923 17:35:44.155327 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:44.155331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:44.155713 (+ 382us) server_negotiation.cc:407] Connection header received
0923 17:35:44.155937 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:44.155941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:44.156009 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:44.156094 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:44.156776 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:44.157494 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:44.158700 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:44.159150 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:44.159289 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:44.163501 (+ 4212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:42563: BlockingRecv error: recv got EOF from 127.20.240.65:42563 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 23 17:35:44 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:44 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:44.601120 30732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:44.592232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:38889)
0923 17:35:44.592759 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:44.592776 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:44.592908 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:44.593216 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:44.593220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:44.593566 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:44.593745 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:44.593755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:44.594970 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:44.594975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:44.595979 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:44.595995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:44.596923 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:44.596931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:44.597123 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:44.597800 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:44.597818 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:44.600717 (+ 2899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":288,"threads_started":1}
W20260923 17:35:44.601192 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:44.592370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:38889 (local address 127.20.240.126:41869)
0923 17:35:44.592556 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:44.592560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:44.592922 (+ 362us) server_negotiation.cc:407] Connection header received
0923 17:35:44.593023 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:44.593028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:44.593104 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:44.593183 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:44.593924 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:44.594744 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:44.596139 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:44.596734 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:44.596839 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:44.600961 (+ 4122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:38889: BlockingRecv error: recv got EOF from 127.20.240.66:38889 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 23 17:35:45 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:45 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:45.059562 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:45.050456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:49081)
0923 17:35:45.051053 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:45.051069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:45.051228 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:45.051577 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:45.051582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:45.051947 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:45.052117 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:45.052127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:45.053101 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:45.053105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:45.054040 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:45.054056 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:45.054767 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:45.054773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:45.054952 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:45.055681 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:45.055702 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:45.059271 (+ 3569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":309,"threads_started":1}
W20260923 17:35:45.059713 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:45.050512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:49081 (local address 127.20.240.126:41869)
0923 17:35:45.050754 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:45.050761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:45.051183 (+ 422us) server_negotiation.cc:407] Connection header received
0923 17:35:45.051351 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:45.051354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:45.051438 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:45.051531 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:45.052267 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:45.052962 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:45.054192 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:45.054621 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:45.054733 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:45.059528 (+ 4795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:49081: BlockingRecv error: recv got EOF from 127.20.240.67:49081 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 23 17:35:45 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:45 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:45.173309 30734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:45.164459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:52577)
0923 17:35:45.164939 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:45.164955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:45.165090 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:45.165473 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:45.165478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:45.165868 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:45.166020 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:45.166029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:45.167020 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:45.167026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:45.168138 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:45.168153 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:45.169153 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:45.169160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:45.169349 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:45.169985 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:45.170009 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:45.173046 (+ 3037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":263,"threads_started":1}
W20260923 17:35:45.173566 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:45.164544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:52577 (local address 127.20.240.126:41869)
0923 17:35:45.164732 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:45.164738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:45.165078 (+ 340us) server_negotiation.cc:407] Connection header received
0923 17:35:45.165217 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:45.165220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:45.165299 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:45.165402 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:45.166162 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:45.166893 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:45.168308 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:45.168991 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:45.169114 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:45.173315 (+ 4201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:52577: BlockingRecv error: recv got EOF from 127.20.240.65:52577 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 17:35:45 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:45 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:45.610445 30735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:45.602255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:38919)
0923 17:35:45.602790 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:45.602809 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:45.602980 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:45.603319 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:45.603324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:45.603713 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:45.603942 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:45.603954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:45.604955 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:45.604961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:45.605939 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:45.605952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:45.606641 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:45.606647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:45.606810 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:45.607425 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:45.607442 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:45.610202 (+ 2760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":257,"threads_started":1}
W20260923 17:35:45.610679 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:45.602577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:38919 (local address 127.20.240.126:41869)
0923 17:35:45.602782 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:45.602788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:45.602902 (+ 114us) server_negotiation.cc:407] Connection header received
0923 17:35:45.603099 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:45.603104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:45.603190 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:45.603275 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:45.604090 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:45.604810 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:45.606079 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:45.606504 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:45.606610 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:45.610408 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:38919: BlockingRecv error: recv got EOF from 127.20.240.66:38919 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 23 17:35:46 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:46 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:46.069511 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:46.060913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:36367)
0923 17:35:46.061388 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:46.061405 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:46.061529 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:46.061896 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:46.061900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:46.062272 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:46.062454 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:46.062464 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:46.063627 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:46.063631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:46.064631 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:46.064644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:46.065354 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:46.065358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:46.065481 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:46.066058 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:46.066074 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:46.069272 (+ 3198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":256,"threads_started":1}
W20260923 17:35:46.069702 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:46.060925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:36367 (local address 127.20.240.126:41869)
0923 17:35:46.061137 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:46.061145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:46.061539 (+ 394us) server_negotiation.cc:407] Connection header received
0923 17:35:46.061617 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:46.061621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:46.061711 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:46.061827 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:46.062609 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:46.063485 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:46.064758 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:46.065208 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:46.065304 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:46.069469 (+ 4165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:36367: BlockingRecv error: recv got EOF from 127.20.240.67:36367 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 17:35:46 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:46 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:46.183053 30737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:46.174652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:47799)
0923 17:35:46.175221 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:46.175241 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:46.175400 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:46.175728 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:46.175733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:46.176156 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:46.176319 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:46.176328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:46.177394 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:46.177399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:46.178336 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:46.178349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:46.179082 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:46.179087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:46.179289 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:46.179879 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:46.179899 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:46.182710 (+ 2811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":246,"threads_started":1}
W20260923 17:35:46.183221 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:46.174736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:47799 (local address 127.20.240.126:41869)
0923 17:35:46.175033 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:46.175040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:46.175314 (+ 274us) server_negotiation.cc:407] Connection header received
0923 17:35:46.175522 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:46.175528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:46.175598 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:46.175680 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:46.176487 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:46.177241 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:46.178482 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:46.178942 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:46.179051 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:46.182979 (+ 3928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:47799: BlockingRecv error: recv got EOF from 127.20.240.65:47799 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 17:35:46 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:46 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:46.620450 30738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:46.611529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:35529)
0923 17:35:46.612093 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:46.612114 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:46.612250 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:46.612632 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:46.612638 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:46.613171 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:46.613416 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:46.613432 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:46.614762 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:46.614767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:46.615779 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:46.615794 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:46.616492 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:46.616498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:46.616638 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:46.617274 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:46.617292 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:46.620117 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":266,"threads_started":1}
W20260923 17:35:46.620652 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:46.611800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:35529 (local address 127.20.240.126:41869)
0923 17:35:46.612019 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:46.612024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:46.612201 (+ 177us) server_negotiation.cc:407] Connection header received
0923 17:35:46.612365 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:46.612371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:46.612449 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:46.612570 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:46.613591 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:46.614611 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:46.615918 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:46.616338 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:46.616435 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:46.620372 (+ 3937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:35529: BlockingRecv error: recv got EOF from 127.20.240.66:35529 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 23 17:35:47 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:47 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:47.080240 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:47.070549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:50573)
0923 17:35:47.071059 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:47.071075 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:47.071221 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:47.071629 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:47.071634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:47.072061 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:47.072242 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:47.072252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:47.073448 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:47.073453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:47.074416 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:47.074431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:47.075436 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:47.075441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:47.075588 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:47.076386 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:47.076414 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:47.079963 (+ 3549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":244,"threads_started":1}
W20260923 17:35:47.080401 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:47.070686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:50573 (local address 127.20.240.126:41869)
0923 17:35:47.070910 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:47.070917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:47.071199 (+ 282us) server_negotiation.cc:407] Connection header received
0923 17:35:47.071340 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:47.071345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:47.071445 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:47.071550 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:47.072421 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:47.073254 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:47.074591 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:47.075264 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:47.075381 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:47.080171 (+ 4790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:50573: BlockingRecv error: recv got EOF from 127.20.240.67:50573 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 23 17:35:47 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:47 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:47.192909 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:47.184191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:51559)
0923 17:35:47.184667 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:47.184683 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:47.184864 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:47.185240 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:47.185244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:47.185683 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:47.185927 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:47.185939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:47.187095 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:47.187099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:47.188160 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:47.188174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:47.188941 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:47.188946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:47.189069 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:47.189747 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:47.189769 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:47.192676 (+ 2907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":249,"threads_started":1}
W20260923 17:35:47.193280 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:47.184282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:51559 (local address 127.20.240.126:41869)
0923 17:35:47.184531 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:47.184538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:47.184801 (+ 263us) server_negotiation.cc:407] Connection header received
0923 17:35:47.185004 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:47.185009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:47.185075 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:47.185163 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:47.186118 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:47.186960 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:47.188298 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:47.188718 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:47.188857 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:47.193004 (+ 4147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:51559: BlockingRecv error: recv got EOF from 127.20.240.65:51559 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 23 17:35:47 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:47 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:47.631130 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:47.621575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:38457)
0923 17:35:47.622140 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:47.622160 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:47.622282 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:47.622614 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:47.622617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:47.623011 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:47.623203 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:47.623217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:47.624675 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:47.624681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:47.626039 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:47.626059 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:47.626989 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:47.626995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:47.627194 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:47.627803 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:47.627825 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:47.630896 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":330,"threads_started":1}
W20260923 17:35:47.631459 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:47.621706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:38457 (local address 127.20.240.126:41869)
0923 17:35:47.621873 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:47.621880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:47.622256 (+ 376us) server_negotiation.cc:407] Connection header received
0923 17:35:47.622401 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:47.622407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:47.622485 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:47.622605 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:47.623355 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:47.624476 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:47.626233 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:47.626807 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:47.626950 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:47.631191 (+ 4241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:38457: BlockingRecv error: recv got EOF from 127.20.240.66:38457 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 23 17:35:48 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:48 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:48.096462 30742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:48.081470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:34849)
0923 17:35:48.082159 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:48.082176 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:48.082331 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:48.082810 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:48.082815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:48.083368 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:48.083543 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:48.083554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:48.084844 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:48.084849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:48.085877 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:48.085891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:48.086859 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:48.086868 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:48.087022 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:48.092898 (+ 5876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:48.092919 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:48.096156 (+ 3237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":403,"threads_started":1}
W20260923 17:35:48.096665 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:48.081624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:34849 (local address 127.20.240.126:41869)
0923 17:35:48.081868 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:48.081875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:48.082298 (+ 423us) server_negotiation.cc:407] Connection header received
0923 17:35:48.082493 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:48.082517 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:48.082647 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:48.082746 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:48.083696 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:48.084636 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:48.086067 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:48.086617 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:48.086718 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:48.096399 (+ 9681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:34849: BlockingRecv error: recv got EOF from 127.20.240.67:34849 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 23 17:35:48 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:48 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:48.204443 30743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:48.194132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:36153)
0923 17:35:48.194738 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:48.194755 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:48.194915 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:48.195431 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:48.195435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:48.195954 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:48.196183 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:48.196197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:48.197663 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:48.197670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:48.199086 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:48.199103 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:48.200036 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:48.200042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:48.200209 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:48.200869 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:48.200895 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:48.204088 (+ 3193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":266,"threads_started":1}
W20260923 17:35:48.204571 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:48.194228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:36153 (local address 127.20.240.126:41869)
0923 17:35:48.194420 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:48.194427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:48.194860 (+ 433us) server_negotiation.cc:407] Connection header received
0923 17:35:48.195050 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:48.195056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:48.195147 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:48.195263 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:48.196350 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:48.197337 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:48.199263 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:48.199873 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:48.200006 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:48.204354 (+ 4348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:36153: BlockingRecv error: recv got EOF from 127.20.240.65:36153 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 17:35:48 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:48 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:48.641325 30744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:48.632023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:40141)
0923 17:35:48.632522 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:48.632538 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:48.632661 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:48.633231 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:48.633237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:48.633687 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:48.633868 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:48.633878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:48.635108 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:48.635113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:48.636131 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:48.636147 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:48.636849 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:48.636853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:48.637016 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:48.637654 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:48.637672 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:48.640984 (+ 3312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":227,"threads_started":1}
W20260923 17:35:48.641364 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:48.632077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:40141 (local address 127.20.240.126:41869)
0923 17:35:48.632291 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:48.632298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:48.632636 (+ 338us) server_negotiation.cc:407] Connection header received
0923 17:35:48.632917 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:48.632921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:48.632989 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:48.633081 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:48.634074 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:48.634968 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:48.636311 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:48.636705 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:48.636801 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:48.641129 (+ 4328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:40141: BlockingRecv error: recv got EOF from 127.20.240.66:40141 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 23 17:35:49 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:49 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:49.106884 30745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:49.097548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:48221)
0923 17:35:49.098148 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:49.098191 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:49.098340 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:49.098740 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:49.098745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:49.099402 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:49.099625 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:49.099638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:49.100822 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:49.100826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:49.101789 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:49.101802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:49.102610 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:49.102615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:49.102744 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:49.103408 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:49.103426 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:49.106613 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":267,"threads_started":1}
W20260923 17:35:49.107262 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:49.097599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:48221 (local address 127.20.240.126:41869)
0923 17:35:49.097785 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:49.097792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:49.098283 (+ 491us) server_negotiation.cc:407] Connection header received
0923 17:35:49.098469 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:49.098475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:49.098556 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:49.098668 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:49.099857 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:49.100662 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:49.101958 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:49.102439 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:49.102542 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:49.106973 (+ 4431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:48221: BlockingRecv error: recv got EOF from 127.20.240.67:48221 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 23 17:35:49 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:49 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:49.214470 30746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:49.205576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:53785)
0923 17:35:49.206100 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:49.206116 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:49.206251 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:49.206627 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:49.206631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:49.207064 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:49.207294 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:49.207307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:49.208416 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:49.208421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:49.209341 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:49.209354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:49.210130 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:49.210137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:49.210291 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:49.210909 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:49.210927 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:49.214136 (+ 3209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":264,"threads_started":1}
W20260923 17:35:49.214674 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:49.205929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:53785 (local address 127.20.240.126:41869)
0923 17:35:49.206111 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:49.206116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:49.206233 (+ 117us) server_negotiation.cc:407] Connection header received
0923 17:35:49.206369 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:49.206373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:49.206445 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:49.206538 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:49.207482 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:49.208234 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:49.209513 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:49.209939 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:49.210055 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:49.214415 (+ 4360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:53785: BlockingRecv error: recv got EOF from 127.20.240.65:53785 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 23 17:35:49 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:49 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:49.652215 30747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:49.642486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:38803)
0923 17:35:49.643092 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:49.643109 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:49.643264 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:49.643570 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:49.643573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:49.644034 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:49.644197 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:49.644210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:49.645699 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:49.645704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:49.646750 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:49.646773 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:49.647781 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:49.647791 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:49.647985 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:49.648733 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:49.648752 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:49.651804 (+ 3052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":319,"threads_started":1}
W20260923 17:35:49.652285 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:49.642644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:38803 (local address 127.20.240.126:41869)
0923 17:35:49.642822 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:49.642827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:49.643195 (+ 368us) server_negotiation.cc:407] Connection header received
0923 17:35:49.643372 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:49.643376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:49.643443 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:49.643537 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:49.644436 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:49.645522 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:49.646943 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:49.647510 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:49.647644 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:49.652069 (+ 4425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:38803: BlockingRecv error: recv got EOF from 127.20.240.66:38803 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 17:35:50 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:50 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:50.117003 30748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:50.107901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:37369)
0923 17:35:50.108411 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:50.108427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:50.108549 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:50.109011 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:50.109015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:50.109440 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:50.109600 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:50.109609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:50.110703 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:50.110709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:50.111817 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:50.111832 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:50.112715 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:50.112721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:50.112853 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:50.113474 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:50.113494 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:50.116626 (+ 3132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":258,"threads_started":1}
W20260923 17:35:50.117134 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:50.108066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:37369 (local address 127.20.240.126:41869)
0923 17:35:50.108276 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:50.108280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:50.108497 (+ 217us) server_negotiation.cc:407] Connection header received
0923 17:35:50.108699 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:50.108703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:50.108772 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:50.108861 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:50.109754 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:50.110542 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:50.112014 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:50.112503 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:50.112623 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:50.116865 (+ 4242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:37369: BlockingRecv error: recv got EOF from 127.20.240.67:37369 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 23 17:35:50 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:50 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:50.224548 30749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:50.215669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:49841)
0923 17:35:50.216296 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:50.216315 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:50.216473 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:50.216862 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:50.216866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:50.217341 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:50.217574 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:50.217588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:50.218681 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:50.218686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:50.219937 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:50.219952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:50.220716 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:50.220721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:50.220901 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:50.221532 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:50.221550 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:50.224295 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":328,"threads_started":1}
W20260923 17:35:50.224742 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:50.215773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:49841 (local address 127.20.240.126:41869)
0923 17:35:50.216097 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:50.216104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:50.216397 (+ 293us) server_negotiation.cc:407] Connection header received
0923 17:35:50.216634 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:50.216638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:50.216707 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:50.216793 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:50.217715 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:50.218535 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:50.220078 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:50.220546 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:50.220639 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:50.224525 (+ 3886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:49841: BlockingRecv error: recv got EOF from 127.20.240.65:49841 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 23 17:35:50 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:50 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:50.662262 30750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:50.653392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:48349)
0923 17:35:50.653901 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:50.653917 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:50.654042 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:50.654341 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:50.654344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:50.654762 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:50.655022 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:50.655033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:50.656380 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:50.656385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:50.657442 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:50.657457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:50.658140 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:50.658144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:50.658268 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:50.658905 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:50.658925 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:50.662015 (+ 3090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":255,"threads_started":1}
W20260923 17:35:50.662425 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:50.653487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:48349 (local address 127.20.240.126:41869)
0923 17:35:50.653664 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:50.653668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:50.654007 (+ 339us) server_negotiation.cc:407] Connection header received
0923 17:35:50.654144 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:50.654148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:50.654216 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:50.654307 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:50.655181 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:50.656227 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:50.657572 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:50.657996 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:50.658086 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:50.662212 (+ 4126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:48349: BlockingRecv error: recv got EOF from 127.20.240.66:48349 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 23 17:35:51 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:51 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:51.126883 30751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:51.118077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:35093)
0923 17:35:51.118576 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:51.118593 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:51.118722 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:51.119059 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:51.119063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:51.119572 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:51.119748 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:51.119816 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:51.120987 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:51.120992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:51.121974 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:51.121987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:51.122859 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:51.122867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:51.123023 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:51.123569 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:51.123587 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:51.126601 (+ 3014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":257,"threads_started":1}
W20260923 17:35:51.127210 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:51.118188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:35093 (local address 127.20.240.126:41869)
0923 17:35:51.118358 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:51.118363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:51.118665 (+ 302us) server_negotiation.cc:407] Connection header received
0923 17:35:51.118828 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:51.118831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:51.118909 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:51.119004 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:51.119981 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:51.120756 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:51.122131 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:51.122611 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:51.122752 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:51.126891 (+ 4139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:35093: BlockingRecv error: recv got EOF from 127.20.240.67:35093 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 17:35:51 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:51 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:51.234589 30752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:51.225483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:39769)
0923 17:35:51.226134 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:51.226151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:51.226266 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:51.226670 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:51.226674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:51.227138 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:51.227338 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:51.227349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:51.228578 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:51.228583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:51.229545 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:51.229558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:51.230337 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:51.230342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:51.230466 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:51.231086 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:51.231103 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:51.234328 (+ 3225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":413,"threads_started":1}
W20260923 17:35:51.234912 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:51.225600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:39769 (local address 127.20.240.126:41869)
0923 17:35:51.225875 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:51.225881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:51.226254 (+ 373us) server_negotiation.cc:407] Connection header received
0923 17:35:51.226449 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:51.226454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:51.226522 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:51.226610 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:51.227509 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:51.228426 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:51.229692 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:51.230192 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:51.230291 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:51.234628 (+ 4337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:39769: BlockingRecv error: recv got EOF from 127.20.240.65:39769 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 23 17:35:51 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:51 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:51.671550 30753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:51.663309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:55175)
0923 17:35:51.663884 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:51.663901 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:51.664013 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:51.664309 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:51.664312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:51.664664 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:51.664816 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:51.664824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:51.665908 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:51.665912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:51.666884 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:51.666897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:51.667608 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:51.667613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:51.667748 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:51.668343 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:51.668378 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:51.671185 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":325,"threads_started":1}
W20260923 17:35:51.671705 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:51.663392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:55175 (local address 127.20.240.126:41869)
0923 17:35:51.663603 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:51.663610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:51.663994 (+ 384us) server_negotiation.cc:407] Connection header received
0923 17:35:51.664114 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:51.664118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:51.664182 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:51.664252 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:51.664993 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:51.665755 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:51.667013 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:51.667464 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:51.667556 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:51.671441 (+ 3885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:55175: BlockingRecv error: recv got EOF from 127.20.240.66:55175 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 17:35:52 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:52 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:52.137144 30754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:52.128048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:56901)
0923 17:35:52.128816 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:52.128836 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:52.128997 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:52.129389 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:52.129395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:52.129876 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:52.130027 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:52.130037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:52.131251 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:52.131256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:52.132225 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:52.132238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:52.133111 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:52.133116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:52.133290 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:52.133915 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:52.133934 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:52.136868 (+ 2934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":530,"threads_started":1}
W20260923 17:35:52.137473 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:52.128212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:56901 (local address 127.20.240.126:41869)
0923 17:35:52.128437 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:52.128442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:52.128952 (+ 510us) server_negotiation.cc:407] Connection header received
0923 17:35:52.129116 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:52.129121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:52.129217 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:52.129336 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:52.130191 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:52.131077 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:52.132421 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:52.132943 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:52.133043 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:52.137181 (+ 4138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:56901: BlockingRecv error: recv got EOF from 127.20.240.67:56901 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 17:35:52 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:52 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:52.244580 30755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:52.235627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:54927)
0923 17:35:52.236193 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:52.236209 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:52.236338 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:52.236666 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:52.236670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:52.237092 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:52.237265 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:52.237273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:52.238516 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:52.238521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:52.239522 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:52.239537 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:52.240441 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:52.240446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:52.240595 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:52.241286 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:52.241311 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:52.244332 (+ 3021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":337,"threads_started":1}
W20260923 17:35:52.244755 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:52.235779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:54927 (local address 127.20.240.126:41869)
0923 17:35:52.235988 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:52.235992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:52.236321 (+ 329us) server_negotiation.cc:407] Connection header received
0923 17:35:52.236450 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:52.236454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:52.236530 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:52.236629 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:52.237471 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:52.238350 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:52.239688 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:52.240261 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:52.240399 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:52.244534 (+ 4135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:54927: BlockingRecv error: recv got EOF from 127.20.240.65:54927 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 23 17:35:52 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:52 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:52.681339 30756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:52.672493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:39793)
0923 17:35:52.673002 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:52.673017 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:52.673153 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:52.673503 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:52.673508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:52.674069 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:52.674272 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:52.674285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:52.675307 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:52.675313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:52.676219 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:52.676232 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:52.676915 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:52.676922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:52.677082 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:52.677663 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:52.677681 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:52.680997 (+ 3316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":277,"threads_started":1}
W20260923 17:35:52.681464 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:52.672599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:39793 (local address 127.20.240.126:41869)
0923 17:35:52.672797 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:52.672803 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:52.673102 (+ 299us) server_negotiation.cc:407] Connection header received
0923 17:35:52.673268 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:52.673290 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:52.673362 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:52.673444 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:52.674426 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:52.675149 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:52.676356 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:52.676769 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:52.676878 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:52.681231 (+ 4353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:39793: BlockingRecv error: recv got EOF from 127.20.240.66:39793 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 23 17:35:53 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:53 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:53.157439 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:53.138204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:47595)
0923 17:35:53.138688 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:53.138704 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:53.138825 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:53.139194 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:53.139197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:53.139616 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:53.139793 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:53.139804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:53.140838 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:53.140842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:53.141739 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:53.141751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:53.142418 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:53.142422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:53.142552 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:53.154028 (+ 11476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:53.154058 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:53.157152 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":265,"threads_started":1}
W20260923 17:35:53.157598 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:53.138261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:47595 (local address 127.20.240.126:41869)
0923 17:35:53.138445 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:53.138452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:53.138785 (+ 333us) server_negotiation.cc:407] Connection header received
0923 17:35:53.138972 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:53.138977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:53.139043 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:53.139127 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:53.139941 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:53.140726 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:53.141874 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:53.142274 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:53.142374 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:53.157405 (+ 15031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:47595: BlockingRecv error: recv got EOF from 127.20.240.67:47595 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 17:35:53 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:53 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:53.254307 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:53.245667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:45815)
0923 17:35:53.246288 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:53.246307 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:53.246435 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:53.246753 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:53.246756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:53.247211 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:53.247378 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:53.247388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:53.248576 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:53.248582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:53.249665 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:53.249679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:53.250426 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:53.250430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:53.250564 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:53.251183 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:53.251210 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:53.254072 (+ 2862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":391,"threads_started":1}
W20260923 17:35:53.254555 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:53.245786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:45815 (local address 127.20.240.126:41869)
0923 17:35:53.246063 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:53.246068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:53.246382 (+ 314us) server_negotiation.cc:407] Connection header received
0923 17:35:53.246527 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:53.246532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:53.246595 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:53.246685 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:53.247540 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:53.248403 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:53.249789 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:53.250267 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:53.250360 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:53.254311 (+ 3951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:45815: BlockingRecv error: recv got EOF from 127.20.240.65:45815 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 23 17:35:53 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:53 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:53.691967 30759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:53.682374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:56667)
0923 17:35:53.682943 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:53.682963 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:53.683137 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:53.683572 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:53.683578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:53.684065 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:53.684279 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:53.684292 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:53.685405 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:53.685412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:53.686469 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:53.686486 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:53.687411 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:53.687419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:53.687617 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:53.688321 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:53.688347 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:53.691645 (+ 3298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":312,"threads_started":1}
W20260923 17:35:53.692186 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:53.682487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:56667 (local address 127.20.240.126:41869)
0923 17:35:53.682828 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:53.682835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:53.683049 (+ 214us) server_negotiation.cc:407] Connection header received
0923 17:35:53.683339 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:53.683343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:53.683415 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:53.683520 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:53.684466 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:53.685233 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:53.686635 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:53.687244 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:53.687381 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:53.691977 (+ 4596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:56667: BlockingRecv error: recv got EOF from 127.20.240.66:56667 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 23 17:35:54 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:54 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:54.167349 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:54.158456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:55875)
0923 17:35:54.159009 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:54.159028 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:54.159183 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:54.159616 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:54.159621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:54.160096 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:54.160255 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:54.160266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:54.161310 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:54.161314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:54.162272 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:54.162284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:54.163080 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:54.163087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:54.163289 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:54.163896 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:54.163915 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:54.167040 (+ 3125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":358,"threads_started":1}
W20260923 17:35:54.167600 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:54.158501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:55875 (local address 127.20.240.126:41869)
0923 17:35:54.158672 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:54.158679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:54.159123 (+ 444us) server_negotiation.cc:407] Connection header received
0923 17:35:54.159298 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:54.159303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:54.159408 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:54.159554 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:54.160383 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:54.161162 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:54.162411 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:54.162923 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:54.163043 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:54.167332 (+ 4289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:55875: BlockingRecv error: recv got EOF from 127.20.240.67:55875 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 23 17:35:54 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:54 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:54.264503 30761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:54.255625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:38241)
0923 17:35:54.256170 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:54.256187 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:54.256403 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:54.256722 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:54.256727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:54.257223 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:54.257460 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:54.257473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:54.258674 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:54.258680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:54.259663 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:54.259676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:54.260467 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:54.260472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:54.260598 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:54.261190 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:54.261208 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:54.264221 (+ 3013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":294,"threads_started":1}
W20260923 17:35:54.264741 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:54.255761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:38241 (local address 127.20.240.126:41869)
0923 17:35:54.256043 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:54.256050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:54.256283 (+ 233us) server_negotiation.cc:407] Connection header received
0923 17:35:54.256511 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:54.256515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:54.256579 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:54.256660 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:54.257641 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:54.258519 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:54.259797 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:54.260270 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:54.260371 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:54.264508 (+ 4137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:38241: BlockingRecv error: recv got EOF from 127.20.240.65:38241 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 23 17:35:54 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:54 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:54.701977 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:54.693019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:58263)
0923 17:35:54.693570 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:54.693592 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:54.693736 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:54.694205 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:54.694210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:54.694652 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:54.694811 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:54.694821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:54.695986 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:54.695990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:54.696925 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:54.696938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:54.697654 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:54.697660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:54.697805 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:54.698449 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:54.698477 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:54.701717 (+ 3240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":305,"threads_started":1}
W20260923 17:35:54.702240 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:54.693297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:58263 (local address 127.20.240.126:41869)
0923 17:35:54.693544 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:54.693551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:54.693678 (+ 127us) server_negotiation.cc:407] Connection header received
0923 17:35:54.693869 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:54.693873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:54.693976 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:54.694172 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:54.694971 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:54.695843 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:54.697061 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:54.697472 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:54.697577 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:54.701998 (+ 4421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:58263: BlockingRecv error: recv got EOF from 127.20.240.66:58263 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 23 17:35:55 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:55 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:55.177237 30763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:55.168417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:56185)
0923 17:35:55.169029 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:55.169046 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:55.169161 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:55.169487 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:55.169491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:55.169913 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:55.170097 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:55.170110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:55.171242 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:55.171246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:55.172291 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:55.172305 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:55.173093 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:55.173098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:55.173231 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:55.173854 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:55.173872 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:55.176967 (+ 3095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":264,"threads_started":1}
W20260923 17:35:55.177496 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:55.168556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:56185 (local address 127.20.240.126:41869)
0923 17:35:55.168776 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:55.168782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:55.169118 (+ 336us) server_negotiation.cc:407] Connection header received
0923 17:35:55.169248 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:55.169252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:55.169318 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:55.169401 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:55.170256 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:55.171063 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:55.172443 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:55.172907 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:55.173003 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:55.177246 (+ 4243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:56185: BlockingRecv error: recv got EOF from 127.20.240.67:56185 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 23 17:35:55 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:55 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:55.274101 30764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:55.265745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:45105)
0923 17:35:55.266426 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:55.266446 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:55.266578 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:55.266898 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:55.266911 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:55.267378 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:55.267557 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:55.267567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:55.268593 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:55.268598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:55.269512 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:55.269524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:55.270211 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:55.270217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:55.270384 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:55.270965 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:55.270984 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:55.273786 (+ 2802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":438,"threads_started":1}
W20260923 17:35:55.274194 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:55.265898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:45105 (local address 127.20.240.126:41869)
0923 17:35:55.266183 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:55.266187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:55.266533 (+ 346us) server_negotiation.cc:407] Connection header received
0923 17:35:55.266689 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:55.266695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:55.266760 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:55.266846 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:55.267699 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:55.268449 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:55.269651 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:55.270075 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:55.270180 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:55.273988 (+ 3808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:45105: BlockingRecv error: recv got EOF from 127.20.240.65:45105 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
Sep 23 17:35:55 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:55 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:55.711318 30765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:55.702936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:36679)
0923 17:35:55.703438 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:55.703459 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:55.703614 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:55.703977 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:55.703982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:55.704432 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:55.704625 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:55.704636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:55.705728 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:55.705732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:55.706645 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:55.706657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:55.707411 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:55.707416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:55.707547 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:55.708162 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:55.708181 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:55.710995 (+ 2814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"spinlock_wait_cycles":48896,"thread_start_us":361,"threads_started":1}
W20260923 17:35:55.711585 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:55.703089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:36679 (local address 127.20.240.126:41869)
0923 17:35:55.703301 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:55.703308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:55.703547 (+ 239us) server_negotiation.cc:407] Connection header received
0923 17:35:55.703745 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:55.703750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:55.703818 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:55.703907 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:55.704788 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:55.705589 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:55.706797 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:55.707248 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:55.707344 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:55.711354 (+ 4010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:36679: BlockingRecv error: recv got EOF from 127.20.240.66:36679 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 17:35:56 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:56 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.67@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:56.187479 30766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:56.178364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.67:48143)
0923 17:35:56.178851 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:56.178868 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:56.178988 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:56.179396 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:56.179400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:56.179856 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:56.180058 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:56.180069 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:56.181441 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:56.181446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:56.182543 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:56.182557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:56.183485 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:56.183490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:56.183624 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:56.184266 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:56.184292 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:56.187205 (+ 2913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":238,"threads_started":1}
W20260923 17:35:56.187649 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:56.178590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:48143 (local address 127.20.240.126:41869)
0923 17:35:56.178909 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:56.178917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:56.178939 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:35:56.179109 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:56.179113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:56.179223 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:56.179347 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:56.180237 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:56.181285 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:56.182672 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:56.183343 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:56.183454 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:56.187436 (+ 3982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.67:48143: BlockingRecv error: recv got EOF from 127.20.240.67:48143 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 23 17:35:56 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:56 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.65@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:56.284482 30767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:56.275388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.65:53575)
0923 17:35:56.275951 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:56.275969 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:56.276096 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:56.276435 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:56.276439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:56.276916 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:56.277102 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:56.277112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:56.278199 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:56.278203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:56.279303 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:56.279319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:56.280114 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:56.280118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:56.280249 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:56.280899 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:56.280925 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:56.284202 (+ 3277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":247,"threads_started":1}
W20260923 17:35:56.284682 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:56.275485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:53575 (local address 127.20.240.126:41869)
0923 17:35:56.275720 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:56.275726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:56.276053 (+ 327us) server_negotiation.cc:407] Connection header received
0923 17:35:56.276209 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:56.276212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:56.276278 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:56.276361 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:56.277251 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:56.278024 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:56.279479 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:56.279933 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:56.280045 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:56.284474 (+ 4429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.65:53575: BlockingRecv error: recv got EOF from 127.20.240.65:53575 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 23 17:35:56 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:35:56 dist-test-slave-ppjl krb5kdc[29721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.240.66@KRBTEST.COM for oryx/127.20.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:35:56.721925 30768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:56.712457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41869 (local address 127.20.240.66:48451)
0923 17:35:56.713001 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:56.713017 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:56.713125 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:56.713436 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:56.713440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:56.713927 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:56.714093 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:56.714103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:56.715373 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:56.715378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:56.716310 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:56.716323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:56.717427 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:56.717432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:56.717569 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:56.718311 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:35:56.718333 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:56.721605 (+ 3272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.240.126:41869: Server oryx/127.20.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":294,"threads_started":1}
W20260923 17:35:56.722301 30459 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:56.712536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:48451 (local address 127.20.240.126:41869)
0923 17:35:56.712776 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:56.712781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:56.713093 (+ 312us) server_negotiation.cc:407] Connection header received
0923 17:35:56.713232 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:56.713236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:56.713304 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:56.713391 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:56.714241 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:56.715233 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:56.716467 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:56.717230 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:56.717360 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:56.721936 (+ 4576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.240.66:48451: BlockingRecv error: recv got EOF from 127.20.240.66:48451 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:35:56.997572 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 30309
I20260923 17:35:57.010018 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:56.998688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.20.240.65:41441)
0923 17:35:56.999253 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:56.999260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:56.999287 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:35:56.999352 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:56.999358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:56.999607 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:56.999765 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:57.001303 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:57.002406 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:57.003868 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:57.004444 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:57.004556 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:57.007077 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:57.007086 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:57.007100 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:57.007135 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:57.009519 (+ 2384us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.240.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 17:35:57.009777 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44378: Request ticket server kudu/127.20.240.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":244,"threads_started":1}
W20260923 17:35:57.010113 30770 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44378: Request ticket server kudu/127.20.240.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 17:35:57.010267 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 30309: Not authorized: Client connection negotiation failed: client connection to 127.20.240.65:41441: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.240.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 17:35:57.010474 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 30309
I20260923 17:35:57.024142 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 30461
I20260923 17:35:57.037086 30771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:57.025337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.20.240.66:37549)
0923 17:35:57.025986 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:57.025993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:57.026030 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:35:57.026090 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:57.026097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:57.026392 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:57.026581 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:57.027934 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:57.029035 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:57.030283 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:57.030845 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:57.030961 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:57.033721 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:57.033731 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:57.033747 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:57.033789 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:57.036438 (+ 2649us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.240.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 17:35:57.036776 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48478: Request ticket server kudu/127.20.240.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":334,"threads_started":1}
W20260923 17:35:57.037186 30771 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48478: Request ticket server kudu/127.20.240.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 17:35:57.037213 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 30461: Not authorized: Client connection negotiation failed: client connection to 127.20.240.66:37549: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.240.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 17:35:57.037369 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 30461
I20260923 17:35:57.050431 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 30592
I20260923 17:35:57.063180 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:57.051495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.20.240.67:45625)
0923 17:35:57.052059 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:57.052066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:57.052093 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:35:57.052161 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:57.052167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:57.052401 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:57.052556 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:57.053914 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:57.054980 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:57.056239 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:57.056845 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:57.056998 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:57.060018 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:57.060028 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:57.060044 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:35:57.060085 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:57.062614 (+ 2529us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.240.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 17:35:57.062905 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41354: Request ticket server kudu/127.20.240.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":256,"threads_started":1}
W20260923 17:35:57.063287 30772 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41354: Request ticket server kudu/127.20.240.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 17:35:57.063609 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 30592: Not authorized: Client connection negotiation failed: client connection to 127.20.240.67:45625: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.240.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 17:35:57.063774 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 30592
I20260923 17:35:57.076591 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 30240
I20260923 17:35:57.078014 30302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:57.196189 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 30240
2026-09-23T17:35:57Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20975 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T17:35:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:35:57Z Disabled control of system clock
I20260923 17:35:57.268214 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:42797
--webserver_interface=127.20.240.126
--webserver_port=0
--builtin_ntp_servers=127.20.240.84:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:42797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 17:35:57.596410 30777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:57.596735 30777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:57.596824 30777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:57.604450 30777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:35:57.604580 30777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:57.604617 30777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:57.604681 30777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:35:57.604732 30777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:35:57.614202 30777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:42797
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:42797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:57.615545 30777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:57.617311 30777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:57.629614 30783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:57.629679 30785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:57.629734 30782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:57.630121 30777 server_base.cc:1034] running on GCE node
I20260923 17:35:57.631284 30777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:57.632789 30777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:57.634034 30777 hybrid_clock.cc:648] HybridClock initialized: now 1790184957633960 us; error 53 us; skew 500 ppm
I20260923 17:35:57.634418 30777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:35:57.637156 30777 webserver.cc:460] Webserver started at http://127.20.240.126:39443/ using document root <none> and password file <none>
I20260923 17:35:57.637961 30777 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:57.638065 30777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:57.638407 30777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:57.641706 30777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "eb7a4f8739c341288e0827922d842a8b"
format_stamp: "Formatted at 2026-09-23 17:35:57 on dist-test-slave-ppjl"
server_key: "8efffb2b98a4e9e02df27b92a514df04"
server_key_iv: "876511a65c11775299d8c34fd4533a4f"
server_key_version: "encryptionkey@0"
I20260923 17:35:57.642530 30777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "eb7a4f8739c341288e0827922d842a8b"
format_stamp: "Formatted at 2026-09-23 17:35:57 on dist-test-slave-ppjl"
server_key: "8efffb2b98a4e9e02df27b92a514df04"
server_key_iv: "876511a65c11775299d8c34fd4533a4f"
server_key_version: "encryptionkey@0"
I20260923 17:35:57.648088 30777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:35:57.652058 30791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:57.653584 30777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:35:57.653776 30777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "eb7a4f8739c341288e0827922d842a8b"
format_stamp: "Formatted at 2026-09-23 17:35:57 on dist-test-slave-ppjl"
server_key: "8efffb2b98a4e9e02df27b92a514df04"
server_key_iv: "876511a65c11775299d8c34fd4533a4f"
server_key_version: "encryptionkey@0"
I20260923 17:35:57.654000 30777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:57.674227 30777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:57.675333 30777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:57.675637 30777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:57.693552 30777 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:42797
I20260923 17:35:57.693557 30842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:42797 every 8 connection(s)
I20260923 17:35:57.694978 30777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:35:57.698827 30843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:35:57.701010 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 30777
I20260923 17:35:57.701172 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:35:57.701870 21441 external_mini_cluster.cc:1428] Setting key a4d5d101b28ec3ca07d851b88f3ef52e
I20260923 17:35:57.712445 30843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b: Bootstrap starting.
I20260923 17:35:57.716374 30843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b: Neither blocks nor log segments found. Creating new log.
I20260923 17:35:57.717772 30843 log.cc:824] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:35:57.718183 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:57.704023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48794 (local address 127.20.240.126:42797)
0923 17:35:57.705153 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:57.705170 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:57.705226 (+ 56us) server_negotiation.cc:407] Connection header received
0923 17:35:57.707584 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:57.707614 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:57.708250 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:57.708731 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:57.709435 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:57.711413 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:57.712877 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:57.713894 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:57.714085 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:57.715232 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:57.715243 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:57.715270 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:57.715991 (+ 721us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:57.716118 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:57.716146 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:57.716276 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:57.716514 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 17:35:57.716887 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":339,"threads_started":1}
I20260923 17:35:57.721827 30843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b: No bootstrap required, opened a new log
I20260923 17:35:57.729417 30843 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7a4f8739c341288e0827922d842a8b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42797 } }
I20260923 17:35:57.729823 30843 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:35:57.729921 30843 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7a4f8739c341288e0827922d842a8b, State: Initialized, Role: FOLLOWER
I20260923 17:35:57.730487 30843 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7a4f8739c341288e0827922d842a8b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42797 } }
I20260923 17:35:57.730710 30843 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:35:57.730834 30843 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:35:57.730990 30843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:35:57.733439 30843 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7a4f8739c341288e0827922d842a8b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42797 } }
I20260923 17:35:57.733959 30843 leader_election.cc:302] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb7a4f8739c341288e0827922d842a8b; no voters:
I20260923 17:35:57.734555 30843 leader_election.cc:288] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:35:57.734694 30848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:35:57.735415 30848 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [term 1 LEADER]: Becoming Leader. State: Replica: eb7a4f8739c341288e0827922d842a8b, State: Running, Role: LEADER
I20260923 17:35:57.735924 30848 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7a4f8739c341288e0827922d842a8b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42797 } }
I20260923 17:35:57.736649 30843 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:35:57.738783 30850 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7a4f8739c341288e0827922d842a8b. Latest consensus state: current_term: 1 leader_uuid: "eb7a4f8739c341288e0827922d842a8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7a4f8739c341288e0827922d842a8b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42797 } } }
I20260923 17:35:57.739199 30850 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:57.740499 30849 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb7a4f8739c341288e0827922d842a8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb7a4f8739c341288e0827922d842a8b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42797 } } }
I20260923 17:35:57.740832 30849 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b [sys.catalog]: This master's current role is: LEADER
I20260923 17:35:57.741369 30857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:35:57.747308 30857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:35:57.757722 30857 catalog_manager.cc:1347] Generated new cluster ID: 71a0204f88094239993981c02033efc5
I20260923 17:35:57.757853 30857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:35:57.776789 30857 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:35:57.778247 30857 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:35:57.785614 30857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb7a4f8739c341288e0827922d842a8b: Generated new TSK 0
I20260923 17:35:57.786540 30857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:35:57.792430 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42797
--builtin_ntp_servers=127.20.240.84:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 17:35:58.148411 30867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:58.148705 30867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:58.148792 30867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:58.156296 30867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:58.156411 30867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:58.156546 30867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:35:58.165946 30867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:58.167343 30867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:58.169180 30867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:58.181975 30875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:58.182168 30873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:58.182511 30872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:58.183076 30867 server_base.cc:1034] running on GCE node
I20260923 17:35:58.183708 30867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:58.184692 30867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:58.185937 30867 hybrid_clock.cc:648] HybridClock initialized: now 1790184958185847 us; error 54 us; skew 500 ppm
I20260923 17:35:58.186261 30867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:35:58.191776 30867 webserver.cc:460] Webserver started at http://127.20.240.65:44691/ using document root <none> and password file <none>
I20260923 17:35:58.192536 30867 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:58.192646 30867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:58.192973 30867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:58.195946 30867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "4ce5cb779be645959dc9cc54ab9f6d7a"
format_stamp: "Formatted at 2026-09-23 17:35:58 on dist-test-slave-ppjl"
server_key: "dfceb80496c0c2415ccba073a7636670"
server_key_iv: "a374d7b99c4a17394c1851e44b745069"
server_key_version: "encryptionkey@0"
I20260923 17:35:58.196736 30867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "4ce5cb779be645959dc9cc54ab9f6d7a"
format_stamp: "Formatted at 2026-09-23 17:35:58 on dist-test-slave-ppjl"
server_key: "dfceb80496c0c2415ccba073a7636670"
server_key_iv: "a374d7b99c4a17394c1851e44b745069"
server_key_version: "encryptionkey@0"
I20260923 17:35:58.202217 30867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:35:58.205981 30881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:58.207489 30867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:35:58.207646 30867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "4ce5cb779be645959dc9cc54ab9f6d7a"
format_stamp: "Formatted at 2026-09-23 17:35:58 on dist-test-slave-ppjl"
server_key: "dfceb80496c0c2415ccba073a7636670"
server_key_iv: "a374d7b99c4a17394c1851e44b745069"
server_key_version: "encryptionkey@0"
I20260923 17:35:58.207849 30867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:58.227941 30867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:58.228827 30867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:58.229123 30867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:58.230113 30867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:58.231622 30867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:58.231722 30867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:58.231828 30867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:58.231896 30867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:58.273370 30867 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:34181
I20260923 17:35:58.273450 30993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:34181 every 8 connection(s)
I20260923 17:35:58.274950 30867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:35:58.278915 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 30867
I20260923 17:35:58.279095 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:35:58.279793 21441 external_mini_cluster.cc:1428] Setting key f5e4922ebceae86b76e18a598d494c5a
I20260923 17:35:58.282878 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42797
--builtin_ntp_servers=127.20.240.84:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 17:35:58.290391 30996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:58.277873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:38731 (local address 127.20.240.126:42797)
0923 17:35:58.278288 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:58.278293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:58.280007 (+ 1714us) server_negotiation.cc:407] Connection header received
0923 17:35:58.282217 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:58.282224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:58.282316 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:58.282432 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:58.283961 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:58.285114 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:58.287502 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:58.287948 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:58.288047 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:58.289623 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:58.289629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:58.289633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:58.289655 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:58.289704 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:58.289716 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:58.289791 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:58.290026 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 17:35:58.290113 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":216,"threads_started":1}
I20260923 17:35:58.291635 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:58.278676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:42797 (local address 127.20.240.65:38731)
0923 17:35:58.279832 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:58.279878 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:58.281940 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:58.282704 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:58.282716 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:58.282853 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:58.283769 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:58.283798 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:58.285502 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:58.285508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:58.287342 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:58.287367 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:58.288149 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:58.288155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:58.288458 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:58.289213 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:35:58.289216 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:58.289273 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:35:58.289378 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:58.289808 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:58.289818 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:58.289825 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:35:58.290001 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 17:35:58.290425 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":367,"threads_started":1}
I20260923 17:35:58.293183 30994 heartbeater.cc:347] Connected to a master server at 127.20.240.126:42797
I20260923 17:35:58.293483 30994 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:58.294288 30994 heartbeater.cc:511] Master 127.20.240.126:42797 requested a full tablet report, sending...
I20260923 17:35:58.296590 30808 ts_manager.cc:194] Registered new tserver with Master: 4ce5cb779be645959dc9cc54ab9f6d7a (127.20.240.65:34181)
I20260923 17:35:58.298050 30808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.65:38731
W20260923 17:35:58.610009 30998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:58.610308 30998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:58.610419 30998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:58.618003 30998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:58.618120 30998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:58.618258 30998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:35:58.627583 30998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:58.628886 30998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:58.630688 30998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:58.644055 31003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:58.644246 31004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:58.644392 31006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:58.645852 30998 server_base.cc:1034] running on GCE node
I20260923 17:35:58.646516 30998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:58.647629 30998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:58.648829 30998 hybrid_clock.cc:648] HybridClock initialized: now 1790184958648773 us; error 43 us; skew 500 ppm
I20260923 17:35:58.649214 30998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:35:58.651712 30998 webserver.cc:460] Webserver started at http://127.20.240.66:43603/ using document root <none> and password file <none>
I20260923 17:35:58.652459 30998 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:58.652557 30998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:58.652875 30998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:58.655941 30998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "88845e8d742048f1b5e086fd780a5515"
format_stamp: "Formatted at 2026-09-23 17:35:58 on dist-test-slave-ppjl"
server_key: "7ffb68d0a5866b3f12b58a66a52a4ac2"
server_key_iv: "2f0e990db831cbd4ed650869f95f16ea"
server_key_version: "encryptionkey@0"
I20260923 17:35:58.656769 30998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "88845e8d742048f1b5e086fd780a5515"
format_stamp: "Formatted at 2026-09-23 17:35:58 on dist-test-slave-ppjl"
server_key: "7ffb68d0a5866b3f12b58a66a52a4ac2"
server_key_iv: "2f0e990db831cbd4ed650869f95f16ea"
server_key_version: "encryptionkey@0"
I20260923 17:35:58.662333 30998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:35:58.666038 31012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:58.667668 30998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:35:58.667837 30998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "88845e8d742048f1b5e086fd780a5515"
format_stamp: "Formatted at 2026-09-23 17:35:58 on dist-test-slave-ppjl"
server_key: "7ffb68d0a5866b3f12b58a66a52a4ac2"
server_key_iv: "2f0e990db831cbd4ed650869f95f16ea"
server_key_version: "encryptionkey@0"
I20260923 17:35:58.668038 30998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:58.699522 30998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:58.700583 30998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:58.700909 30998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:58.702103 30998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:58.703963 30998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:58.704071 30998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:58.704167 30998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:58.704233 30998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:58.737696 30998 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:36221
I20260923 17:35:58.737767 31124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:36221 every 8 connection(s)
I20260923 17:35:58.739056 30998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:35:58.747879 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 30998
I20260923 17:35:58.748039 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:35:58.748503 21441 external_mini_cluster.cc:1428] Setting key 55d142fa8fac4115389fa04c8f0060e8
I20260923 17:35:58.751515 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42797
--builtin_ntp_servers=127.20.240.84:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 17:35:58.752593 30996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:58.741782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:59161 (local address 127.20.240.126:42797)
0923 17:35:58.741961 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:58.741966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:58.743656 (+ 1690us) server_negotiation.cc:407] Connection header received
0923 17:35:58.745592 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:58.745597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:58.745670 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:58.745787 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:58.747074 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:58.747895 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:58.749757 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:58.750180 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:58.750301 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:58.751764 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:58.751771 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:58.751775 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:58.751798 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:58.751840 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:58.751853 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:58.751931 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:58.752163 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 17:35:58.752280 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:35:58.753762 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:58.742294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:42797 (local address 127.20.240.66:59161)
0923 17:35:58.743475 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:58.743528 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:58.745295 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:58.746099 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:58.746109 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:58.746209 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:58.746883 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:58.746902 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:58.748062 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:58.748066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:58.749612 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:58.749627 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:58.750418 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:58.750427 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:58.750656 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:58.751409 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:35:58.751412 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:58.751458 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:35:58.751554 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:58.751947 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:58.751960 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:58.751969 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:35:58.752136 (+ 167us) client_negotiation.cc:240] Negotiation successful
0923 17:35:58.752583 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":268,"threads_started":1}
I20260923 17:35:58.755393 31125 heartbeater.cc:347] Connected to a master server at 127.20.240.126:42797
I20260923 17:35:58.755741 31125 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:58.756513 31125 heartbeater.cc:511] Master 127.20.240.126:42797 requested a full tablet report, sending...
I20260923 17:35:58.758081 30808 ts_manager.cc:194] Registered new tserver with Master: 88845e8d742048f1b5e086fd780a5515 (127.20.240.66:36221)
I20260923 17:35:58.758936 30808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.66:59161
W20260923 17:35:59.089709 31128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:35:59.090049 31128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:35:59.090161 31128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:35:59.097626 31128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:35:59.097762 31128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:35:59.097898 31128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:35:59.107179 31128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:35:59.108529 31128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:35:59.110351 31128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:35:59.123557 31133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:59.123557 31134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:35:59.123944 31136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:35:59.125065 31128 server_base.cc:1034] running on GCE node
I20260923 17:35:59.125725 31128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:35:59.126788 31128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:35:59.128010 31128 hybrid_clock.cc:648] HybridClock initialized: now 1790184959127953 us; error 48 us; skew 500 ppm
I20260923 17:35:59.128341 31128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:35:59.130743 31128 webserver.cc:460] Webserver started at http://127.20.240.67:39773/ using document root <none> and password file <none>
I20260923 17:35:59.131506 31128 fs_manager.cc:359] Metadata directory not provided
I20260923 17:35:59.131613 31128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:35:59.131939 31128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:35:59.134863 31128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "126461e0149b4ecc8fbe0cd1ceb39132"
format_stamp: "Formatted at 2026-09-23 17:35:59 on dist-test-slave-ppjl"
server_key: "ede7ec23a836423d81eed2009349839f"
server_key_iv: "ff1726de7dec229e6d92bbf3fc5f1509"
server_key_version: "encryptionkey@0"
I20260923 17:35:59.135681 31128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "126461e0149b4ecc8fbe0cd1ceb39132"
format_stamp: "Formatted at 2026-09-23 17:35:59 on dist-test-slave-ppjl"
server_key: "ede7ec23a836423d81eed2009349839f"
server_key_iv: "ff1726de7dec229e6d92bbf3fc5f1509"
server_key_version: "encryptionkey@0"
I20260923 17:35:59.141222 31128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:35:59.144867 31142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:59.146273 31128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:35:59.146448 31128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "126461e0149b4ecc8fbe0cd1ceb39132"
format_stamp: "Formatted at 2026-09-23 17:35:59 on dist-test-slave-ppjl"
server_key: "ede7ec23a836423d81eed2009349839f"
server_key_iv: "ff1726de7dec229e6d92bbf3fc5f1509"
server_key_version: "encryptionkey@0"
I20260923 17:35:59.146664 31128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:35:59.171332 31128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:35:59.172274 31128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:35:59.172559 31128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:35:59.173554 31128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:35:59.175029 31128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:35:59.175130 31128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:59.175258 31128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:35:59.175330 31128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:35:59.206940 31128 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:40969
I20260923 17:35:59.207011 31254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:40969 every 8 connection(s)
I20260923 17:35:59.208218 31128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:35:59.216049 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 31128
I20260923 17:35:59.216228 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:35:59.216755 21441 external_mini_cluster.cc:1428] Setting key c7cdc609821c6817abc4f82ab963a9b5
I20260923 17:35:59.221702 30996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:59.210846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:53353 (local address 127.20.240.126:42797)
0923 17:35:59.211028 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:59.211032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:59.212437 (+ 1405us) server_negotiation.cc:407] Connection header received
0923 17:35:59.214187 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:59.214193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:59.214272 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:59.214360 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:59.215439 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:59.216339 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:59.218729 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:59.219443 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:59.219543 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:59.221027 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:59.221032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:59.221036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:59.221054 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:59.221087 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:59.221099 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:59.221167 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:59.221358 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 17:35:59.221446 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:35:59.222764 31257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:59.211401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:42797 (local address 127.20.240.67:53353)
0923 17:35:59.212260 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0923 17:35:59.212296 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 17:35:59.213927 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:35:59.214588 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:35:59.214596 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:35:59.214686 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:35:59.215277 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:59.215296 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:59.216541 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:59.216546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:59.218517 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:35:59.218533 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:35:59.219664 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:35:59.219672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:35:59.219990 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:59.220711 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:35:59.220714 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:35:59.220756 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:35:59.220852 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:35:59.221183 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:35:59.221191 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:35:59.221197 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:35:59.221348 (+ 151us) client_negotiation.cc:240] Negotiation successful
0923 17:35:59.221667 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":255,"threads_started":1}
I20260923 17:35:59.224040 31255 heartbeater.cc:347] Connected to a master server at 127.20.240.126:42797
I20260923 17:35:59.224351 31255 heartbeater.cc:464] Registering TS with master...
I20260923 17:35:59.225165 31255 heartbeater.cc:511] Master 127.20.240.126:42797 requested a full tablet report, sending...
I20260923 17:35:59.226469 30808 ts_manager.cc:194] Registered new tserver with Master: 126461e0149b4ecc8fbe0cd1ceb39132 (127.20.240.67:40969)
I20260923 17:35:59.227375 30808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.67:53353
I20260923 17:35:59.232743 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 17:35:59.238627 30996 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:35:59.237094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48806 (local address 127.20.240.126:42797)
0923 17:35:59.237398 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:59.237404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:59.237762 (+ 358us) server_negotiation.cc:407] Connection header received
0923 17:35:59.237873 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:59.237877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:59.237953 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:59.238040 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:59.238385 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48806: BlockingRecv error: recv got EOF from 127.0.0.1:48806 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
I20260923 17:35:59.239528 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 30867
I20260923 17:35:59.247115 31264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:59.240406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49350 (local address 127.20.240.65:34181)
0923 17:35:59.240939 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:59.240946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:59.240970 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:35:59.241024 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:59.241030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:59.241233 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:59.241368 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:59.241773 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:59.242771 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:59.244051 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:59.244610 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:59.244722 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:59.245697 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:59.245706 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:59.245719 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:59.246293 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:59.246340 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:59.246360 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:59.246469 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:59.246611 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 17:35:59.246764 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":241,"threads_started":1}
I20260923 17:35:59.248485 30989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:59.328961 30994 heartbeater.cc:503] Master 127.20.240.126:42797 was elected leader, sending a full tablet report...
I20260923 17:35:59.330255 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 30867
W20260923 17:35:59.343287 30792 connection.cc:535] server connection from 127.20.240.65:38731 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 17:35:59.343557 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 30998
I20260923 17:35:59.351796 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:59.344382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.20.240.66:36221)
0923 17:35:59.344936 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:59.344944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:59.344971 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:35:59.345028 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:59.345035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:59.345243 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:59.345374 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:59.345756 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:59.346774 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:59.348125 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:59.348750 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:59.348864 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:59.349982 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:59.349991 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:59.350012 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:59.350698 (+ 686us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:59.350751 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:59.350774 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:59.350870 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:59.351146 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 17:35:59.351387 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":278,"threads_started":1}
I20260923 17:35:59.353125 31120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:59.434222 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 30998
I20260923 17:35:59.447242 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 31128
I20260923 17:35:59.457044 31268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:35:59.448528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35764 (local address 127.20.240.67:40969)
0923 17:35:59.449365 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0923 17:35:59.449372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:35:59.449405 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:35:59.449477 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:35:59.449483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:35:59.449715 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:35:59.449856 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:35:59.450382 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:59.451721 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:59.453144 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:35:59.453790 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:35:59.453903 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:35:59.455101 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:35:59.455111 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:35:59.455126 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:35:59.455865 (+ 739us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:35:59.455996 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:35:59.456028 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:35:59.456135 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:35:59.456445 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 17:35:59.456615 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":522,"threads_started":1}
I20260923 17:35:59.458480 31250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:59.543200 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 31128
I20260923 17:35:59.555817 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 30777
I20260923 17:35:59.557159 30838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:35:59.660291 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 30777
2026-09-23T17:35:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2460 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T17:35:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:35:59Z Disabled control of system clock
I20260923 17:35:59.728124 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:42565
--webserver_interface=127.20.240.126
--webserver_port=0
--builtin_ntp_servers=127.20.240.84:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:42565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 17:36:00.056504 31274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:36:00.056839 31274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:36:00.056927 31274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:36:00.064379 31274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:36:00.064509 31274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:36:00.064548 31274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:36:00.064604 31274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:36:00.064647 31274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:36:00.074014 31274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:42565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:42565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:36:00.075326 31274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:36:00.077072 31274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:36:00.088814 31279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:00.088879 31282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:00.088913 31280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:36:00.089673 31274 server_base.cc:1034] running on GCE node
I20260923 17:36:00.090420 31274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:36:00.091877 31274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:36:00.093078 31274 hybrid_clock.cc:648] HybridClock initialized: now 1790184960093030 us; error 45 us; skew 500 ppm
I20260923 17:36:00.093436 31274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:36:00.095991 31274 webserver.cc:460] Webserver started at http://127.20.240.126:45815/ using document root <none> and password file <none>
I20260923 17:36:00.096722 31274 fs_manager.cc:359] Metadata directory not provided
I20260923 17:36:00.096839 31274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:36:00.097151 31274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:36:00.100184 31274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "5ea98b61362549b3b5aaacb0f3aa96cd"
format_stamp: "Formatted at 2026-09-23 17:36:00 on dist-test-slave-ppjl"
server_key: "76fe4a7167993c4b727e13417552ca09"
server_key_iv: "c61d05e7b53c2bbbdb6cdaefcaf4797b"
server_key_version: "encryptionkey@0"
I20260923 17:36:00.101024 31274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "5ea98b61362549b3b5aaacb0f3aa96cd"
format_stamp: "Formatted at 2026-09-23 17:36:00 on dist-test-slave-ppjl"
server_key: "76fe4a7167993c4b727e13417552ca09"
server_key_iv: "c61d05e7b53c2bbbdb6cdaefcaf4797b"
server_key_version: "encryptionkey@0"
I20260923 17:36:00.106266 31274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:36:00.109895 31288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:00.111373 31274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:36:00.111537 31274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "5ea98b61362549b3b5aaacb0f3aa96cd"
format_stamp: "Formatted at 2026-09-23 17:36:00 on dist-test-slave-ppjl"
server_key: "76fe4a7167993c4b727e13417552ca09"
server_key_iv: "c61d05e7b53c2bbbdb6cdaefcaf4797b"
server_key_version: "encryptionkey@0"
I20260923 17:36:00.111739 31274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:36:00.117470 31274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:36:00.118247 31274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:36:00.118513 31274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:36:00.137041 31274 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:42565
I20260923 17:36:00.137058 31339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:42565 every 8 connection(s)
I20260923 17:36:00.138516 31274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:36:00.141940 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 31274
I20260923 17:36:00.142161 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:36:00.142942 21441 external_mini_cluster.cc:1428] Setting key 5cd4605b4db316615854396b5f78e023
I20260923 17:36:00.143705 31340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:36:00.152268 31343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:00.145351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49320 (local address 127.20.240.126:42565)
0923 17:36:00.146451 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:00.146463 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:00.146511 (+ 48us) server_negotiation.cc:407] Connection header received
0923 17:36:00.148641 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:00.148667 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:00.149138 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:00.149571 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:00.150608 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:00.150618 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:00.150632 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:36:00.150649 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:00.150712 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:36:00.150729 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:00.150947 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:00.151090 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 17:36:00.151376 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":455,"threads_started":1}
I20260923 17:36:00.157622 31340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd: Bootstrap starting.
I20260923 17:36:00.161373 31340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd: Neither blocks nor log segments found. Creating new log.
I20260923 17:36:00.162544 31340 log.cc:824] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd: Log is configured to *not* fsync() on all Append() calls
I20260923 17:36:00.165935 31340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd: No bootstrap required, opened a new log
I20260923 17:36:00.173800 31340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea98b61362549b3b5aaacb0f3aa96cd" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42565 } }
I20260923 17:36:00.174237 31340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:36:00.174335 31340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ea98b61362549b3b5aaacb0f3aa96cd, State: Initialized, Role: FOLLOWER
I20260923 17:36:00.174926 31340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea98b61362549b3b5aaacb0f3aa96cd" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42565 } }
I20260923 17:36:00.175203 31340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:36:00.175347 31340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:36:00.175513 31340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:36:00.177764 31340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea98b61362549b3b5aaacb0f3aa96cd" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42565 } }
I20260923 17:36:00.178339 31340 leader_election.cc:302] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ea98b61362549b3b5aaacb0f3aa96cd; no voters:
I20260923 17:36:00.178999 31340 leader_election.cc:288] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:36:00.179152 31345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:36:00.179875 31345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [term 1 LEADER]: Becoming Leader. State: Replica: 5ea98b61362549b3b5aaacb0f3aa96cd, State: Running, Role: LEADER
I20260923 17:36:00.180454 31345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea98b61362549b3b5aaacb0f3aa96cd" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42565 } }
I20260923 17:36:00.181223 31340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:36:00.183516 31346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ea98b61362549b3b5aaacb0f3aa96cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea98b61362549b3b5aaacb0f3aa96cd" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42565 } } }
I20260923 17:36:00.183712 31347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ea98b61362549b3b5aaacb0f3aa96cd. Latest consensus state: current_term: 1 leader_uuid: "5ea98b61362549b3b5aaacb0f3aa96cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ea98b61362549b3b5aaacb0f3aa96cd" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42565 } } }
I20260923 17:36:00.184088 31346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [sys.catalog]: This master's current role is: LEADER
I20260923 17:36:00.184092 31347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd [sys.catalog]: This master's current role is: LEADER
I20260923 17:36:00.185755 31353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:36:00.190647 31353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:36:00.199596 31353 catalog_manager.cc:1347] Generated new cluster ID: 939ce31350454101b9e0d144831b7b4f
I20260923 17:36:00.199744 31353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:36:00.214401 31353 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:36:00.214826 31353 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:36:00.228603 31353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ea98b61362549b3b5aaacb0f3aa96cd: Generated new TSK 0
I20260923 17:36:00.229528 31353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:36:00.237725 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42565
--builtin_ntp_servers=127.20.240.84:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 17:36:00.583581 31364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:36:00.583868 31364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:36:00.583995 31364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:36:00.591464 31364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:36:00.591573 31364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:36:00.591727 31364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:36:00.601305 31364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:36:00.602604 31364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:36:00.604466 31364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:36:00.617627 31369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:00.617594 31372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:00.617664 31370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:36:00.618783 31364 server_base.cc:1034] running on GCE node
I20260923 17:36:00.619421 31364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:36:00.620321 31364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:36:00.621534 31364 hybrid_clock.cc:648] HybridClock initialized: now 1790184960621484 us; error 36 us; skew 500 ppm
I20260923 17:36:00.621870 31364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:36:00.624096 31364 webserver.cc:460] Webserver started at http://127.20.240.65:38775/ using document root <none> and password file <none>
I20260923 17:36:00.624799 31364 fs_manager.cc:359] Metadata directory not provided
I20260923 17:36:00.624895 31364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:36:00.625216 31364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:36:00.629527 31364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "df819f2f5ce943e2aa9c1c0dd957b761"
format_stamp: "Formatted at 2026-09-23 17:36:00 on dist-test-slave-ppjl"
server_key: "cd839ebcc320ab89210c4a1cb1a82aeb"
server_key_iv: "88264fe0be001245ef501ec8a518d665"
server_key_version: "encryptionkey@0"
I20260923 17:36:00.630334 31364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "df819f2f5ce943e2aa9c1c0dd957b761"
format_stamp: "Formatted at 2026-09-23 17:36:00 on dist-test-slave-ppjl"
server_key: "cd839ebcc320ab89210c4a1cb1a82aeb"
server_key_iv: "88264fe0be001245ef501ec8a518d665"
server_key_version: "encryptionkey@0"
I20260923 17:36:00.635582 31364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:36:00.639343 31378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:00.640983 31364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:36:00.641161 31364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "df819f2f5ce943e2aa9c1c0dd957b761"
format_stamp: "Formatted at 2026-09-23 17:36:00 on dist-test-slave-ppjl"
server_key: "cd839ebcc320ab89210c4a1cb1a82aeb"
server_key_iv: "88264fe0be001245ef501ec8a518d665"
server_key_version: "encryptionkey@0"
I20260923 17:36:00.641390 31364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:36:00.647295 31364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:36:00.648175 31364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:36:00.648430 31364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:36:00.649374 31364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:36:00.650866 31364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:36:00.650965 31364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:00.651072 31364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:36:00.651139 31364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:00.681396 31364 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:43993
I20260923 17:36:00.681473 31490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:43993 every 8 connection(s)
I20260923 17:36:00.682878 31364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:36:00.691442 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:00.685567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:47747 (local address 127.20.240.126:42565)
0923 17:36:00.686036 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:00.686040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:00.687398 (+ 1358us) server_negotiation.cc:407] Connection header received
0923 17:36:00.689168 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:00.689174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:00.689246 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:00.689343 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:00.690681 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:00.690687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:00.690690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:36:00.690707 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:00.690750 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:36:00.690763 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:00.690871 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:00.691062 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 17:36:00.691132 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":224,"threads_started":1}
I20260923 17:36:00.692447 31494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:00.686265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:42565 (local address 127.20.240.65:47747)
0923 17:36:00.687238 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:00.687283 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:00.688878 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:00.689611 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:00.689619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:00.689729 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:36:00.690361 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:36:00.690364 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:36:00.690412 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:36:00.690502 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:36:00.690852 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:36:00.690862 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:36:00.690870 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:36:00.691132 (+ 262us) client_negotiation.cc:240] Negotiation successful
0923 17:36:00.691443 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":260,"threads_started":1}
I20260923 17:36:00.693303 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 31364
I20260923 17:36:00.693483 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:36:00.693985 31491 heartbeater.cc:347] Connected to a master server at 127.20.240.126:42565
I20260923 17:36:00.694193 21441 external_mini_cluster.cc:1428] Setting key e7a9b496e90a81a30b2660369b8200c1
I20260923 17:36:00.694339 31491 heartbeater.cc:464] Registering TS with master...
I20260923 17:36:00.695037 31491 heartbeater.cc:511] Master 127.20.240.126:42565 requested a full tablet report, sending...
I20260923 17:36:00.697451 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42565
--builtin_ntp_servers=127.20.240.84:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 17:36:00.697553 31305 ts_manager.cc:194] Registered new tserver with Master: df819f2f5ce943e2aa9c1c0dd957b761 (127.20.240.65:43993)
W20260923 17:36:01.022920 31495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:36:01.023262 31495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:36:01.023386 31495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:36:01.030860 31495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:36:01.030974 31495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:36:01.031119 31495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:36:01.040470 31495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:36:01.041778 31495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:36:01.043601 31495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:36:01.057531 31500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:01.057533 31503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:01.057499 31501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:36:01.058225 31495 server_base.cc:1034] running on GCE node
I20260923 17:36:01.058936 31495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:36:01.060031 31495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:36:01.061295 31495 hybrid_clock.cc:648] HybridClock initialized: now 1790184961061218 us; error 70 us; skew 500 ppm
I20260923 17:36:01.061676 31495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:36:01.064325 31495 webserver.cc:460] Webserver started at http://127.20.240.66:44017/ using document root <none> and password file <none>
I20260923 17:36:01.065070 31495 fs_manager.cc:359] Metadata directory not provided
I20260923 17:36:01.065176 31495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:36:01.065491 31495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:36:01.068521 31495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "32ade804a2d640698de037bb9e5ee8f9"
format_stamp: "Formatted at 2026-09-23 17:36:01 on dist-test-slave-ppjl"
server_key: "c10143dbab1b1095ad61433568c3f478"
server_key_iv: "51dcd88e11a3674e5ff27977346b6008"
server_key_version: "encryptionkey@0"
I20260923 17:36:01.069406 31495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "32ade804a2d640698de037bb9e5ee8f9"
format_stamp: "Formatted at 2026-09-23 17:36:01 on dist-test-slave-ppjl"
server_key: "c10143dbab1b1095ad61433568c3f478"
server_key_iv: "51dcd88e11a3674e5ff27977346b6008"
server_key_version: "encryptionkey@0"
I20260923 17:36:01.074896 31495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:36:01.078590 31509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:01.080099 31495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:36:01.080261 31495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "32ade804a2d640698de037bb9e5ee8f9"
format_stamp: "Formatted at 2026-09-23 17:36:01 on dist-test-slave-ppjl"
server_key: "c10143dbab1b1095ad61433568c3f478"
server_key_iv: "51dcd88e11a3674e5ff27977346b6008"
server_key_version: "encryptionkey@0"
I20260923 17:36:01.080493 31495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:36:01.086220 31495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:36:01.087029 31495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:36:01.087330 31495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:36:01.088272 31495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:36:01.089748 31495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:36:01.089845 31495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:01.089962 31495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:36:01.090030 31495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:01.121289 31495 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:35763
I20260923 17:36:01.121337 31621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:35763 every 8 connection(s)
I20260923 17:36:01.122706 31495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:36:01.130568 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 31495
I20260923 17:36:01.130759 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:36:01.130990 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:01.125408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:37267 (local address 127.20.240.126:42565)
0923 17:36:01.125632 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:01.125636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:01.127124 (+ 1488us) server_negotiation.cc:407] Connection header received
0923 17:36:01.128809 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:01.128815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:01.128891 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:01.128978 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:01.130264 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:01.130270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:01.130275 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:36:01.130292 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:01.130325 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:36:01.130337 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:01.130444 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:01.130630 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 17:36:01.130700 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 17:36:01.131304 21441 external_mini_cluster.cc:1428] Setting key eb2b69f181313abf874b691f42e9de52
I20260923 17:36:01.132026 31624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:01.125976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:42565 (local address 127.20.240.66:37267)
0923 17:36:01.126954 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:01.126996 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:01.128535 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:01.129232 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:01.129241 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:01.129337 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:36:01.129964 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:36:01.129966 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:36:01.130005 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:36:01.130092 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:36:01.130426 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:36:01.130436 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:36:01.130444 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:36:01.130702 (+ 258us) client_negotiation.cc:240] Negotiation successful
0923 17:36:01.130971 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":305,"threads_started":1}
I20260923 17:36:01.133391 31622 heartbeater.cc:347] Connected to a master server at 127.20.240.126:42565
I20260923 17:36:01.133683 31622 heartbeater.cc:464] Registering TS with master...
I20260923 17:36:01.134097 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42565
--builtin_ntp_servers=127.20.240.84:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 17:36:01.134368 31622 heartbeater.cc:511] Master 127.20.240.126:42565 requested a full tablet report, sending...
I20260923 17:36:01.135977 31304 ts_manager.cc:194] Registered new tserver with Master: 32ade804a2d640698de037bb9e5ee8f9 (127.20.240.66:35763)
W20260923 17:36:01.465938 31625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:36:01.466235 31625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:36:01.466356 31625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:36:01.473838 31625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:36:01.473960 31625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:36:01.474104 31625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:36:01.484043 31625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:42565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:36:01.485567 31625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:36:01.487542 31625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:36:01.501268 31630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:01.501205 31633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:01.501180 31631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:36:01.502107 31625 server_base.cc:1034] running on GCE node
I20260923 17:36:01.502703 31625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:36:01.503772 31625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:36:01.504992 31625 hybrid_clock.cc:648] HybridClock initialized: now 1790184961504939 us; error 46 us; skew 500 ppm
I20260923 17:36:01.505321 31625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:36:01.507750 31625 webserver.cc:460] Webserver started at http://127.20.240.67:40423/ using document root <none> and password file <none>
I20260923 17:36:01.508443 31625 fs_manager.cc:359] Metadata directory not provided
I20260923 17:36:01.508545 31625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:36:01.508908 31625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:36:01.512276 31625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "fed6f5fae8554435b64591fbdce51c15"
format_stamp: "Formatted at 2026-09-23 17:36:01 on dist-test-slave-ppjl"
server_key: "da78d4338c7109b87cad73126d3b8f79"
server_key_iv: "795761f20149ef51f974e822547cf34b"
server_key_version: "encryptionkey@0"
I20260923 17:36:01.513181 31625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "fed6f5fae8554435b64591fbdce51c15"
format_stamp: "Formatted at 2026-09-23 17:36:01 on dist-test-slave-ppjl"
server_key: "da78d4338c7109b87cad73126d3b8f79"
server_key_iv: "795761f20149ef51f974e822547cf34b"
server_key_version: "encryptionkey@0"
I20260923 17:36:01.518666 31625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:36:01.522739 31639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:01.524447 31625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:36:01.524624 31625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "fed6f5fae8554435b64591fbdce51c15"
format_stamp: "Formatted at 2026-09-23 17:36:01 on dist-test-slave-ppjl"
server_key: "da78d4338c7109b87cad73126d3b8f79"
server_key_iv: "795761f20149ef51f974e822547cf34b"
server_key_version: "encryptionkey@0"
I20260923 17:36:01.524863 31625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:36:01.530848 31625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:36:01.531710 31625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:36:01.531957 31625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:36:01.532860 31625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:36:01.534301 31625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:36:01.534394 31625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:01.534499 31625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:36:01.534565 31625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:01.564934 31625 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:42039
I20260923 17:36:01.564988 31751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:42039 every 8 connection(s)
I20260923 17:36:01.566318 31625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:36:01.566741 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 31625
I20260923 17:36:01.566928 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:36:01.567498 21441 external_mini_cluster.cc:1428] Setting key f052fe19a65b2392568759384711a553
I20260923 17:36:01.574895 31493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:01.569321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:40501 (local address 127.20.240.126:42565)
0923 17:36:01.569517 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:01.569523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:01.571188 (+ 1665us) server_negotiation.cc:407] Connection header received
0923 17:36:01.572842 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:01.572846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:01.572912 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:01.572993 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:01.574222 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:01.574227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:01.574230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:36:01.574246 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:01.574279 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:36:01.574291 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:01.574391 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:01.574556 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 17:36:01.574612 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:36:01.575747 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:01.569964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:42565 (local address 127.20.240.67:40501)
0923 17:36:01.571011 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:01.571054 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:01.572591 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:01.573176 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:01.573184 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:01.573287 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:36:01.573941 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:36:01.573943 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:36:01.573981 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:36:01.574067 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:36:01.574359 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:36:01.574365 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:36:01.574370 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:36:01.574564 (+ 194us) client_negotiation.cc:240] Negotiation successful
0923 17:36:01.574769 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":306,"threads_started":1}
I20260923 17:36:01.577090 31752 heartbeater.cc:347] Connected to a master server at 127.20.240.126:42565
I20260923 17:36:01.577395 31752 heartbeater.cc:464] Registering TS with master...
I20260923 17:36:01.577987 31752 heartbeater.cc:511] Master 127.20.240.126:42565 requested a full tablet report, sending...
I20260923 17:36:01.579440 31304 ts_manager.cc:194] Registered new tserver with Master: fed6f5fae8554435b64591fbdce51c15 (127.20.240.67:42039)
I20260923 17:36:01.583150 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 17:36:01.589418 31493 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:36:01.587726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40058 (local address 127.20.240.126:42565)
0923 17:36:01.587892 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:01.587896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:01.588400 (+ 504us) server_negotiation.cc:407] Connection header received
0923 17:36:01.588609 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:01.588615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:01.588680 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:01.588764 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:01.589154 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40058: BlockingRecv error: recv got EOF from 127.0.0.1:40058 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:36:01.590078 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 31364
I20260923 17:36:01.593495 31761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:01.590988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.20.240.65:43993)
0923 17:36:01.591540 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:01.591549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:01.591575 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:36:01.591635 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:01.591641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:01.591856 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:01.591987 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:01.592807 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:01.592814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:01.592826 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:36:01.592843 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:01.592946 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:36:01.592963 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:01.593070 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:01.593176 (+ 106us) server_negotiation.cc:299] Negotiation successful
0923 17:36:01.593258 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":292,"threads_started":1}
I20260923 17:36:01.594843 31486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:36:01.675319 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 31364
I20260923 17:36:01.688899 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 31495
I20260923 17:36:01.692512 31763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:01.689776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52814 (local address 127.20.240.66:35763)
0923 17:36:01.690409 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:01.690416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:01.690449 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:36:01.690525 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:01.690532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:01.690768 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:01.690918 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:01.691795 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:01.691803 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:01.691816 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:36:01.691837 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:01.691881 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:36:01.691897 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:01.692012 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:01.692120 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 17:36:01.692244 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":310,"threads_started":1}
I20260923 17:36:01.693794 31617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:36:01.775949 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 31495
I20260923 17:36:01.788899 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 31625
I20260923 17:36:01.792915 31765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:01.789960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.20.240.67:42039)
0923 17:36:01.790593 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:01.790599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:01.790630 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:36:01.790694 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:01.790701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:01.790976 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:01.791111 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:01.792178 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:01.792184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:01.792196 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:36:01.792213 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:01.792251 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:36:01.792265 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:01.792400 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:01.792486 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 17:36:01.792572 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":317,"threads_started":1}
I20260923 17:36:01.794418 31747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:36:01.877521 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 31625
I20260923 17:36:01.891635 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 31274
I20260923 17:36:01.892915 31335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:36:01.998296 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 31274
2026-09-23T17:36:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2337 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:36:02 dist-test-slave-ppjl krb5kdc[31770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:36:02 dist-test-slave-ppjl krb5kdc[31770](info): set up 2 sockets
Sep 23 17:36:02 dist-test-slave-ppjl krb5kdc[31770](info): commencing operation
krb5kdc: starting...
W20260923 17:36:03.028602 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:36:03 dist-test-slave-ppjl krb5kdc[31770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:36:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:36:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:36:03.257000 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:46251
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:46251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:36:03.587070 31783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:36:03.587445 31783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:36:03.587543 31783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:36:03.594858 31783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:36:03.594990 31783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:36:03.595036 31783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:36:03.595093 31783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:36:03.595136 31783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:36:03.605957 31783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:46251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:46251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:36:03.607354 31783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:36:03.608722 31783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:36:03.620925 31788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:03.620886 31789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:03.620857 31791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:36:03.621694 31783 server_base.cc:1034] running on GCE node
I20260923 17:36:03.622494 31783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:36:03.624107 31783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:36:03.625353 31783 hybrid_clock.cc:648] HybridClock initialized: now 1790184963625297 us; error 35 us; skew 500 ppm
I20260923 17:36:03.625783 31783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:36:03 dist-test-slave-ppjl krb5kdc[31770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184963, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:36:03.637058 31783 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:36:03.638932 31783 webserver.cc:460] Webserver started at http://127.20.240.126:40987/ using document root <none> and password file <none>
I20260923 17:36:03.639739 31783 fs_manager.cc:359] Metadata directory not provided
I20260923 17:36:03.639878 31783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:36:03.640210 31783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:36:03.643440 31783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "31db20be1d62413fada1b0c9e05a0a85"
format_stamp: "Formatted at 2026-09-23 17:36:03 on dist-test-slave-ppjl"
server_key: "ee1e9a4eb41bd95abb9abe8d9fad53da"
server_key_iv: "8d83911e9e46fed2ba6529d7e9a96282"
server_key_version: "encryptionkey@0"
I20260923 17:36:03.644207 31783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "31db20be1d62413fada1b0c9e05a0a85"
format_stamp: "Formatted at 2026-09-23 17:36:03 on dist-test-slave-ppjl"
server_key: "ee1e9a4eb41bd95abb9abe8d9fad53da"
server_key_iv: "8d83911e9e46fed2ba6529d7e9a96282"
server_key_version: "encryptionkey@0"
I20260923 17:36:03.649580 31783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:36:03.653244 31798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:03.654789 31783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:36:03.654968 31783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "31db20be1d62413fada1b0c9e05a0a85"
format_stamp: "Formatted at 2026-09-23 17:36:03 on dist-test-slave-ppjl"
server_key: "ee1e9a4eb41bd95abb9abe8d9fad53da"
server_key_iv: "8d83911e9e46fed2ba6529d7e9a96282"
server_key_version: "encryptionkey@0"
I20260923 17:36:03.655244 31783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:36:03.677736 31783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:36:03.682333 31783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:36:03.682621 31783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:36:03.700588 31783 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:46251
I20260923 17:36:03.700623 31849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:46251 every 8 connection(s)
I20260923 17:36:03.702152 31783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:36:03.706182 31850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:36:03.709883 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 31783
I20260923 17:36:03.710114 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:36:03.710738 21441 external_mini_cluster.cc:1428] Setting key c434b0649e31f37091b094a7b58779f0
I20260923 17:36:03.719810 31850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85: Bootstrap starting.
I20260923 17:36:03.723357 31850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85: Neither blocks nor log segments found. Creating new log.
I20260923 17:36:03.724344 31850 log.cc:824] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:36:03 dist-test-slave-ppjl krb5kdc[31770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:36:03.727545 31850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85: No bootstrap required, opened a new log
I20260923 17:36:03.733599 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:03.712796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53702 (local address 127.20.240.126:46251)
0923 17:36:03.713613 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:03.713630 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:03.713684 (+ 54us) server_negotiation.cc:407] Connection header received
0923 17:36:03.715461 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:03.715493 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:03.716062 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:03.716531 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:03.717860 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:03.719844 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:03.721370 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:03.722389 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:03.722565 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:03.726521 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:03.726529 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:03.726543 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:03.726579 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:03.729598 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:03.730400 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:03.730407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:03.730413 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:03.730574 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:03.730940 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:03.730943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:03.730945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:03.731511 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:03.731730 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:03.732167 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 17:36:03.732570 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":344,"threads_started":1}
I20260923 17:36:03.736413 31850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31db20be1d62413fada1b0c9e05a0a85" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 46251 } }
I20260923 17:36:03.736848 31850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:36:03.736949 31850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31db20be1d62413fada1b0c9e05a0a85, State: Initialized, Role: FOLLOWER
I20260923 17:36:03.737515 31850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31db20be1d62413fada1b0c9e05a0a85" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 46251 } }
I20260923 17:36:03.737725 31850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:36:03.737859 31850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:36:03.738003 31850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:36:03.740396 31850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31db20be1d62413fada1b0c9e05a0a85" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 46251 } }
I20260923 17:36:03.740900 31850 leader_election.cc:302] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31db20be1d62413fada1b0c9e05a0a85; no voters:
I20260923 17:36:03.741473 31850 leader_election.cc:288] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:36:03.741592 31855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:36:03.742287 31855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [term 1 LEADER]: Becoming Leader. State: Replica: 31db20be1d62413fada1b0c9e05a0a85, State: Running, Role: LEADER
I20260923 17:36:03.742980 31855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31db20be1d62413fada1b0c9e05a0a85" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 46251 } }
I20260923 17:36:03.743633 31850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:36:03.745704 31857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31db20be1d62413fada1b0c9e05a0a85. Latest consensus state: current_term: 1 leader_uuid: "31db20be1d62413fada1b0c9e05a0a85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31db20be1d62413fada1b0c9e05a0a85" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 46251 } } }
I20260923 17:36:03.745668 31856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31db20be1d62413fada1b0c9e05a0a85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31db20be1d62413fada1b0c9e05a0a85" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 46251 } } }
I20260923 17:36:03.746155 31856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [sys.catalog]: This master's current role is: LEADER
I20260923 17:36:03.748875 31857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85 [sys.catalog]: This master's current role is: LEADER
I20260923 17:36:03.750108 31861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:36:03.755484 31861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:36:03.764308 31861 catalog_manager.cc:1347] Generated new cluster ID: 2de7e00bdf554820afe7dda1c7fbb9b0
I20260923 17:36:03.764423 31861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:36:03.779994 31861 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:36:03.781893 31861 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:36:03.797261 31861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85: Generated new TSK 0
I20260923 17:36:03.798291 31861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:36:03.855545 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:46251
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:36:04.192818 31876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:36:04.193150 31876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:36:04.193285 31876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:36:04.200693 31876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:36:04.200834 31876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:36:04.200984 31876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:36:04.210922 31876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:46251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:36:04.212275 31876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:36:04.213611 31876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:36:04.227479 31881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:04.227423 31884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:04.227459 31882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:36:04.228295 31876 server_base.cc:1034] running on GCE node
I20260923 17:36:04.228940 31876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:36:04.229897 31876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:36:04.231119 31876 hybrid_clock.cc:648] HybridClock initialized: now 1790184964231065 us; error 45 us; skew 500 ppm
I20260923 17:36:04.231489 31876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:36:04 dist-test-slave-ppjl krb5kdc[31770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184964, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:36:04.238600 31876 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:36:04.241611 31876 webserver.cc:460] Webserver started at http://127.20.240.65:43207/ using document root <none> and password file <none>
I20260923 17:36:04.242496 31876 fs_manager.cc:359] Metadata directory not provided
I20260923 17:36:04.242636 31876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:36:04.242950 31876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:36:04.246774 31876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "642bb02c62e44cfaa3c4241856667cce"
format_stamp: "Formatted at 2026-09-23 17:36:04 on dist-test-slave-ppjl"
server_key: "8768b27ca83ab97dac6f5526513c2ecb"
server_key_iv: "00c963d63ad9fb5f77b0e9a535f527df"
server_key_version: "encryptionkey@0"
I20260923 17:36:04.247747 31876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "642bb02c62e44cfaa3c4241856667cce"
format_stamp: "Formatted at 2026-09-23 17:36:04 on dist-test-slave-ppjl"
server_key: "8768b27ca83ab97dac6f5526513c2ecb"
server_key_iv: "00c963d63ad9fb5f77b0e9a535f527df"
server_key_version: "encryptionkey@0"
I20260923 17:36:04.253300 31876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:36:04.256748 31891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:04.258127 31876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:36:04.258262 31876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "642bb02c62e44cfaa3c4241856667cce"
format_stamp: "Formatted at 2026-09-23 17:36:04 on dist-test-slave-ppjl"
server_key: "8768b27ca83ab97dac6f5526513c2ecb"
server_key_iv: "00c963d63ad9fb5f77b0e9a535f527df"
server_key_version: "encryptionkey@0"
I20260923 17:36:04.258428 31876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:36:04.273828 31876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:36:04.278470 31876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:36:04.278739 31876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:36:04.279911 31876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:36:04.281423 31876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:36:04.281509 31876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:04.281595 31876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:36:04.281646 31876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:04.313601 31876 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:35795
I20260923 17:36:04.313635 32003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:35795 every 8 connection(s)
I20260923 17:36:04.314903 31876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:36:04.322264 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 31876
I20260923 17:36:04.322494 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:36:04.323118 21441 external_mini_cluster.cc:1428] Setting key ad4298568210935786457f0c7b1604e1
Sep 23 17:36:04 dist-test-slave-ppjl krb5kdc[31770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184964, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:36:04.336007 32006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:04.317401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:37961 (local address 127.20.240.126:46251)
0923 17:36:04.317857 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:04.317863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:04.318832 (+ 969us) server_negotiation.cc:407] Connection header received
0923 17:36:04.320642 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:04.320647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:04.320716 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:04.320813 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:04.322547 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:04.323429 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:04.325341 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:04.325751 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:04.325846 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:04.329874 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:04.329880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:04.329883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:04.329917 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:04.332502 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:04.333310 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:04.333315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:04.333317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:04.333446 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:04.333894 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:04.333898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:04.333900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:04.334260 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:04.334419 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:04.335450 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0923 17:36:04.335687 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":198,"threads_started":1}
I20260923 17:36:04.337256 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:04.317983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:46251 (local address 127.20.240.65:37961)
0923 17:36:04.318669 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:04.318711 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:04.320383 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:04.321007 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:04.321015 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:04.321558 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:36:04.322340 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:04.322370 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:04.323587 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:04.323592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:04.325171 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:04.325184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:04.325977 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:04.325985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:04.326259 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:04.327047 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:36:04.327076 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:36:04.329618 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:36:04.332676 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:36:04.332687 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:36:04.332710 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:36:04.333174 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:36:04.333559 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:36:04.333564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:36:04.333568 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:36:04.333783 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:36:04.334439 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:36:04.334448 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:36:04.334922 (+ 474us) client_negotiation.cc:769] Sending connection context
0923 17:36:04.335254 (+ 332us) client_negotiation.cc:240] Negotiation successful
0923 17:36:04.335846 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":267,"threads_started":1}
I20260923 17:36:04.339114 32004 heartbeater.cc:347] Connected to a master server at 127.20.240.126:46251
I20260923 17:36:04.339499 32004 heartbeater.cc:464] Registering TS with master...
I20260923 17:36:04.340473 32004 heartbeater.cc:511] Master 127.20.240.126:46251 requested a full tablet report, sending...
I20260923 17:36:04.342851 31815 ts_manager.cc:194] Registered new tserver with Master: 642bb02c62e44cfaa3c4241856667cce (127.20.240.65:35795)
I20260923 17:36:04.344481 31815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:37961
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:36:04.374773 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:46251
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:36:04.700139 32010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:36:04.700440 32010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:36:04.700567 32010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:36:04.708134 32010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:36:04.708257 32010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:36:04.708418 32010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:36:04.718592 32010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:46251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:36:04.720019 32010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:36:04.721398 32010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:36:04.734818 32015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:04.734818 32016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:04.735145 32018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:36:04.735812 32010 server_base.cc:1034] running on GCE node
I20260923 17:36:04.736552 32010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:36:04.737594 32010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:36:04.738891 32010 hybrid_clock.cc:648] HybridClock initialized: now 1790184964738830 us; error 48 us; skew 500 ppm
I20260923 17:36:04.739295 32010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:36:04 dist-test-slave-ppjl krb5kdc[31770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184964, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:36:04.742457 32010 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:36:04.744169 32010 webserver.cc:460] Webserver started at http://127.20.240.66:44505/ using document root <none> and password file <none>
I20260923 17:36:04.744907 32010 fs_manager.cc:359] Metadata directory not provided
I20260923 17:36:04.745024 32010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:36:04.745301 32010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:36:04.748369 32010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "fbfd998697c64da28ea2d7ad50ed2770"
format_stamp: "Formatted at 2026-09-23 17:36:04 on dist-test-slave-ppjl"
server_key: "fe841065f19c3127a0277ab84922fd0a"
server_key_iv: "292eb76858a03058c5483bd6f2d85c7f"
server_key_version: "encryptionkey@0"
I20260923 17:36:04.749140 32010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "fbfd998697c64da28ea2d7ad50ed2770"
format_stamp: "Formatted at 2026-09-23 17:36:04 on dist-test-slave-ppjl"
server_key: "fe841065f19c3127a0277ab84922fd0a"
server_key_iv: "292eb76858a03058c5483bd6f2d85c7f"
server_key_version: "encryptionkey@0"
I20260923 17:36:04.754618 32010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:36:04.758744 32025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:04.760429 32010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:36:04.760591 32010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "fbfd998697c64da28ea2d7ad50ed2770"
format_stamp: "Formatted at 2026-09-23 17:36:04 on dist-test-slave-ppjl"
server_key: "fe841065f19c3127a0277ab84922fd0a"
server_key_iv: "292eb76858a03058c5483bd6f2d85c7f"
server_key_version: "encryptionkey@0"
I20260923 17:36:04.760797 32010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:36:04.796096 32010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:36:04.800976 32010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:36:04.801316 32010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:36:04.802382 32010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:36:04.803929 32010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:36:04.804013 32010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:04.804145 32010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:36:04.804204 32010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:04.837255 32010 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:39103
I20260923 17:36:04.837292 32137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:39103 every 8 connection(s)
I20260923 17:36:04.838603 32010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:36:04.847611 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 32010
I20260923 17:36:04.847795 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:36:04.848243 21441 external_mini_cluster.cc:1428] Setting key d4ae3a4fdbb61b0d8a0d50926308d720
Sep 23 17:36:04 dist-test-slave-ppjl krb5kdc[31770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184964, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:36:04.860148 32140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:04.841301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:55339 (local address 127.20.240.126:46251)
0923 17:36:04.841751 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:04.841755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:04.842812 (+ 1057us) server_negotiation.cc:407] Connection header received
0923 17:36:04.844718 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:04.844724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:04.844813 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:04.844908 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:04.846499 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:04.847348 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:04.849308 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:04.849828 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:04.849936 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:04.853841 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:04.853847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:04.853851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:04.853892 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:04.856920 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:04.857655 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:04.857660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:04.857663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:04.857815 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:04.858246 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:04.858250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:04.858252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:04.858524 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:04.858663 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:04.859499 (+ 836us) server_negotiation.cc:299] Negotiation successful
0923 17:36:04.859750 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":214,"threads_started":1}
I20260923 17:36:04.861195 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:04.841925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:46251 (local address 127.20.240.66:55339)
0923 17:36:04.842605 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:04.842642 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:04.844410 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:04.845183 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:04.845191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:04.845728 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:36:04.846317 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:04.846336 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:04.847513 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:04.847520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:04.849127 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:04.849171 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:04.850063 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:04.850071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:04.850381 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:04.851014 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:36:04.851042 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:36:04.853524 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:36:04.857069 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:36:04.857077 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:36:04.857091 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:36:04.857521 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:36:04.857923 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:36:04.857927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:36:04.857929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:36:04.858118 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:36:04.858763 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:36:04.858769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:36:04.859105 (+ 336us) client_negotiation.cc:769] Sending connection context
0923 17:36:04.859398 (+ 293us) client_negotiation.cc:240] Negotiation successful
0923 17:36:04.859999 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":257,"threads_started":1}
I20260923 17:36:04.862895 32138 heartbeater.cc:347] Connected to a master server at 127.20.240.126:46251
I20260923 17:36:04.863306 32138 heartbeater.cc:464] Registering TS with master...
I20260923 17:36:04.864223 32138 heartbeater.cc:511] Master 127.20.240.126:46251 requested a full tablet report, sending...
I20260923 17:36:04.865926 31815 ts_manager.cc:194] Registered new tserver with Master: fbfd998697c64da28ea2d7ad50ed2770 (127.20.240.66:39103)
I20260923 17:36:04.867038 31815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:55339
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:36:04.900384 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:46251
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:36:05.220216 32144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:36:05.220518 32144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:36:05.220616 32144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:36:05.227901 32144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:36:05.228019 32144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:36:05.228133 32144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:36:05.238106 32144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:46251
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:36:05.239549 32144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:36:05.240922 32144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:36:05.255762 32150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:05.255839 32149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:05.255820 32152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:36:05.256397 32144 server_base.cc:1034] running on GCE node
I20260923 17:36:05.257066 32144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:36:05.258381 32144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:36:05.259588 32144 hybrid_clock.cc:648] HybridClock initialized: now 1790184965259506 us; error 69 us; skew 500 ppm
I20260923 17:36:05.259917 32144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:36:05 dist-test-slave-ppjl krb5kdc[31770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184965, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:36:05.263360 32144 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:36:05.264920 32144 webserver.cc:460] Webserver started at http://127.20.240.67:33395/ using document root <none> and password file <none>
I20260923 17:36:05.265619 32144 fs_manager.cc:359] Metadata directory not provided
I20260923 17:36:05.265730 32144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:36:05.266029 32144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:36:05.268925 32144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "ca57816b5ffe45de8d2c9b3cd974fb04"
format_stamp: "Formatted at 2026-09-23 17:36:05 on dist-test-slave-ppjl"
server_key: "cce5f2dd0745206286808e41661c4ec0"
server_key_iv: "e41f7946aade1a980019b2331a10ddbd"
server_key_version: "encryptionkey@0"
I20260923 17:36:05.269642 32144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "ca57816b5ffe45de8d2c9b3cd974fb04"
format_stamp: "Formatted at 2026-09-23 17:36:05 on dist-test-slave-ppjl"
server_key: "cce5f2dd0745206286808e41661c4ec0"
server_key_iv: "e41f7946aade1a980019b2331a10ddbd"
server_key_version: "encryptionkey@0"
I20260923 17:36:05.274778 32144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:36:05.278667 32159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:05.280249 32144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:36:05.280427 32144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "ca57816b5ffe45de8d2c9b3cd974fb04"
format_stamp: "Formatted at 2026-09-23 17:36:05 on dist-test-slave-ppjl"
server_key: "cce5f2dd0745206286808e41661c4ec0"
server_key_iv: "e41f7946aade1a980019b2331a10ddbd"
server_key_version: "encryptionkey@0"
I20260923 17:36:05.280624 32144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:36:05.310897 32144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:36:05.316212 32144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:36:05.316563 32144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:36:05.317734 32144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:36:05.319458 32144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:36:05.319554 32144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:05.319675 32144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:36:05.319748 32144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:05.348309 32004 heartbeater.cc:503] Master 127.20.240.126:46251 was elected leader, sending a full tablet report...
I20260923 17:36:05.350270 32144 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:42081
I20260923 17:36:05.350369 32271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:42081 every 8 connection(s)
I20260923 17:36:05.351637 32144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:36:05.352730 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 32144
I20260923 17:36:05.352921 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:36:05.353363 21441 external_mini_cluster.cc:1428] Setting key e6cfd8f72d6f0a48acaaa46b4c3664ea
Sep 23 17:36:05 dist-test-slave-ppjl krb5kdc[31770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184965, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:36:05.372568 32140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:05.354527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:53073 (local address 127.20.240.126:46251)
0923 17:36:05.354923 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:05.354929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:05.356142 (+ 1213us) server_negotiation.cc:407] Connection header received
0923 17:36:05.358028 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:05.358032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:05.358111 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:05.358227 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:05.359952 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.360775 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.362437 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.362830 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.362925 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:05.366403 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:05.366408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:05.366412 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:05.366446 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:05.369613 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:05.370319 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:05.370322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:05.370324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:05.370443 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:05.370897 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:05.370901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:05.370903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:05.371259 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:05.371380 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:05.372125 (+ 745us) server_negotiation.cc:299] Negotiation successful
0923 17:36:05.372330 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260923 17:36:05.373588 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:05.355096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:46251 (local address 127.20.240.67:53073)
0923 17:36:05.355976 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:05.356030 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:05.357754 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:05.358388 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:05.358398 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:05.359111 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:36:05.359766 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:05.359786 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.360929 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.360933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:05.362307 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:05.362324 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.363050 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.363056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:05.363330 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:05.363958 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:36:05.363981 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:36:05.366195 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:36:05.369755 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:36:05.369763 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:36:05.369776 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:36:05.370211 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:36:05.370542 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:36:05.370546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:36:05.370548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:36:05.370761 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:36:05.371377 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:36:05.371384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:36:05.371757 (+ 373us) client_negotiation.cc:769] Sending connection context
0923 17:36:05.372009 (+ 252us) client_negotiation.cc:240] Negotiation successful
0923 17:36:05.372455 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":329,"threads_started":1}
I20260923 17:36:05.375074 32272 heartbeater.cc:347] Connected to a master server at 127.20.240.126:46251
I20260923 17:36:05.375398 32272 heartbeater.cc:464] Registering TS with master...
I20260923 17:36:05.376132 32272 heartbeater.cc:511] Master 127.20.240.126:46251 requested a full tablet report, sending...
I20260923 17:36:05.377555 31815 ts_manager.cc:194] Registered new tserver with Master: ca57816b5ffe45de8d2c9b3cd974fb04 (127.20.240.67:42081)
I20260923 17:36:05.378399 31815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:53073
I20260923 17:36:05.381466 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:36:05.399705 32140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:05.385965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.20.240.126:46251)
0923 17:36:05.386127 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:05.386132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:05.386516 (+ 384us) server_negotiation.cc:407] Connection header received
0923 17:36:05.386662 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:05.386666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:05.386736 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:05.386825 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:05.388489 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.389235 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.390596 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.391138 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.391304 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:05.393893 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:05.393898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:05.393901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:05.393938 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:05.396786 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:05.397504 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:05.397508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:05.397510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:05.397725 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:05.398123 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:05.398127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:05.398129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:05.398495 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:05.398709 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:05.399190 (+ 481us) server_negotiation.cc:299] Negotiation successful
0923 17:36:05.399446 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:36:05.404007 31815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:36:05.410916 31815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:36:05.439446 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:05.424046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:39103 (local address 127.0.0.1:55296)
0923 17:36:05.425483 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:05.425527 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:05.425722 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:05.427676 (+ 1954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:05.427686 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:05.427738 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:36:05.428231 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:05.428246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.432141 (+ 3895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.432148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:05.434962 (+ 2814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:05.434987 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.438388 (+ 3401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.438395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:05.438611 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:05.438666 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 17:36:05.438781 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 17:36:05.438979 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1161,"thread_start_us":1003,"threads_started":1}
I20260923 17:36:05.441332 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:05.424284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55296 (local address 127.20.240.66:39103)
0923 17:36:05.426899 (+ 2615us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:05.426908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:05.426939 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:36:05.427069 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:05.427078 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:05.427398 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:05.429870 (+ 2472us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:36:05.430166 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.431924 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.436592 (+ 4668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.438147 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.440465 (+ 2318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:05.440719 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:05.440803 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 17:36:05.440987 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2475,"thread_start_us":2322,"threads_started":1}
I20260923 17:36:05.445331 32073 tablet_service.cc:1467] Processing CreateTablet for tablet 74bf93fa907a41cd875b2139839f1563 (DEFAULT_TABLE table=test-table [id=7eaadcc70ee643ff9db65bcb0d5199f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:36:05.446050 32287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:05.424948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.20.240.65:35795)
0923 17:36:05.426447 (+ 1499us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:05.426470 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:05.435284 (+ 8814us) server_negotiation.cc:407] Connection header received
0923 17:36:05.435856 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:05.435866 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:05.436176 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:05.436382 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:36:05.437472 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.438932 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.443138 (+ 4206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.444688 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.444859 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:05.445123 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:05.445510 (+ 387us) server_negotiation.cc:299] Negotiation successful
0923 17:36:05.445699 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1315,"thread_start_us":1152,"threads_started":1}
I20260923 17:36:05.447038 32073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74bf93fa907a41cd875b2139839f1563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:36:05.447070 32288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:05.424665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:35795 (local address 127.0.0.1:58658)
0923 17:36:05.435121 (+ 10456us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:05.435142 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:05.435633 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:05.436747 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:05.436752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:05.436778 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:36:05.437193 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:05.437254 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.439091 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.439095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:05.442958 (+ 3863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:05.442977 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.444947 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.444955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:05.445257 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:05.445302 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 17:36:05.446481 (+ 1179us) client_negotiation.cc:240] Negotiation successful
0923 17:36:05.446639 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10304,"mutex_wait_us":69,"thread_start_us":15357,"threads_started":1}
I20260923 17:36:05.449147 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:05.426056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:42081 (local address 127.0.0.1:40272)
0923 17:36:05.439766 (+ 13710us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:05.439784 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:05.439879 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:05.442709 (+ 2830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:05.442714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:05.442746 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:36:05.443032 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:05.443045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.445108 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.445113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:05.446933 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:05.446948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.448350 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.448355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:05.448527 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:05.448560 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:36:05.448649 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 17:36:05.448819 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13592,"mutex_wait_us":28,"thread_start_us":9266,"threads_started":1}
I20260923 17:36:05.449292 32289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:05.426770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.20.240.67:42081)
0923 17:36:05.442065 (+ 15295us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:05.442074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:05.442109 (+ 35us) server_negotiation.cc:407] Connection header received
0923 17:36:05.442212 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:05.442220 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:05.442490 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:05.442707 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:36:05.443279 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.444924 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.447085 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.448193 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.448315 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:05.448572 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:05.448765 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 17:36:05.448939 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15115,"thread_start_us":14950,"threads_started":1}
I20260923 17:36:05.450947 31939 tablet_service.cc:1467] Processing CreateTablet for tablet 74bf93fa907a41cd875b2139839f1563 (DEFAULT_TABLE table=test-table [id=7eaadcc70ee643ff9db65bcb0d5199f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:36:05.452487 31939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74bf93fa907a41cd875b2139839f1563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:36:05.455116 32207 tablet_service.cc:1467] Processing CreateTablet for tablet 74bf93fa907a41cd875b2139839f1563 (DEFAULT_TABLE table=test-table [id=7eaadcc70ee643ff9db65bcb0d5199f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:36:05.456704 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74bf93fa907a41cd875b2139839f1563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:36:05.463546 32290 tablet_bootstrap.cc:492] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770: Bootstrap starting.
I20260923 17:36:05.464452 32291 tablet_bootstrap.cc:492] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce: Bootstrap starting.
I20260923 17:36:05.467038 32290 tablet_bootstrap.cc:654] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770: Neither blocks nor log segments found. Creating new log.
I20260923 17:36:05.467942 32291 tablet_bootstrap.cc:654] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce: Neither blocks nor log segments found. Creating new log.
I20260923 17:36:05.468458 32290 log.cc:824] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770: Log is configured to *not* fsync() on all Append() calls
I20260923 17:36:05.469165 32291 log.cc:824] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce: Log is configured to *not* fsync() on all Append() calls
I20260923 17:36:05.472360 32291 tablet_bootstrap.cc:492] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce: No bootstrap required, opened a new log
I20260923 17:36:05.472601 32290 tablet_bootstrap.cc:492] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770: No bootstrap required, opened a new log
I20260923 17:36:05.472860 32291 ts_tablet_manager.cc:1397] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 17:36:05.472896 32290 ts_tablet_manager.cc:1397] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 17:36:05.473050 32294 tablet_bootstrap.cc:492] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04: Bootstrap starting.
I20260923 17:36:05.477159 32294 tablet_bootstrap.cc:654] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04: Neither blocks nor log segments found. Creating new log.
I20260923 17:36:05.478494 32294 log.cc:824] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04: Log is configured to *not* fsync() on all Append() calls
I20260923 17:36:05.480541 32290 raft_consensus.cc:348] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbfd998697c64da28ea2d7ad50ed2770" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 39103 } } peers { permanent_uuid: "ca57816b5ffe45de8d2c9b3cd974fb04" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42081 } } peers { permanent_uuid: "642bb02c62e44cfaa3c4241856667cce" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 35795 } }
I20260923 17:36:05.480998 32290 raft_consensus.cc:374] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:36:05.481106 32290 raft_consensus.cc:729] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbfd998697c64da28ea2d7ad50ed2770, State: Initialized, Role: FOLLOWER
I20260923 17:36:05.481875 32294 tablet_bootstrap.cc:492] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04: No bootstrap required, opened a new log
I20260923 17:36:05.481755 32290 consensus_queue.cc:260] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbfd998697c64da28ea2d7ad50ed2770" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 39103 } } peers { permanent_uuid: "ca57816b5ffe45de8d2c9b3cd974fb04" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42081 } } peers { permanent_uuid: "642bb02c62e44cfaa3c4241856667cce" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 35795 } }
I20260923 17:36:05.482154 32294 ts_tablet_manager.cc:1397] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 17:36:05.483032 32138 heartbeater.cc:503] Master 127.20.240.126:46251 was elected leader, sending a full tablet report...
I20260923 17:36:05.483057 32291 raft_consensus.cc:348] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbfd998697c64da28ea2d7ad50ed2770" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 39103 } } peers { permanent_uuid: "ca57816b5ffe45de8d2c9b3cd974fb04" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42081 } } peers { permanent_uuid: "642bb02c62e44cfaa3c4241856667cce" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 35795 } }
I20260923 17:36:05.483511 32291 raft_consensus.cc:374] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:36:05.483596 32291 raft_consensus.cc:729] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642bb02c62e44cfaa3c4241856667cce, State: Initialized, Role: FOLLOWER
I20260923 17:36:05.483675 32290 ts_tablet_manager.cc:1428] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260923 17:36:05.484153 32291 consensus_queue.cc:260] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbfd998697c64da28ea2d7ad50ed2770" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 39103 } } peers { permanent_uuid: "ca57816b5ffe45de8d2c9b3cd974fb04" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42081 } } peers { permanent_uuid: "642bb02c62e44cfaa3c4241856667cce" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 35795 } }
I20260923 17:36:05.485914 32291 ts_tablet_manager.cc:1428] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260923 17:36:05.490944 32294 raft_consensus.cc:348] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbfd998697c64da28ea2d7ad50ed2770" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 39103 } } peers { permanent_uuid: "ca57816b5ffe45de8d2c9b3cd974fb04" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42081 } } peers { permanent_uuid: "642bb02c62e44cfaa3c4241856667cce" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 35795 } }
I20260923 17:36:05.491384 32294 raft_consensus.cc:374] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:36:05.491477 32294 raft_consensus.cc:729] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca57816b5ffe45de8d2c9b3cd974fb04, State: Initialized, Role: FOLLOWER
I20260923 17:36:05.492059 32294 consensus_queue.cc:260] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbfd998697c64da28ea2d7ad50ed2770" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 39103 } } peers { permanent_uuid: "ca57816b5ffe45de8d2c9b3cd974fb04" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42081 } } peers { permanent_uuid: "642bb02c62e44cfaa3c4241856667cce" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 35795 } }
I20260923 17:36:05.493199 32272 heartbeater.cc:503] Master 127.20.240.126:46251 was elected leader, sending a full tablet report...
I20260923 17:36:05.493882 32294 ts_tablet_manager.cc:1428] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260923 17:36:05.567333 32005 tablet.cc:2369] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:36:05.590739 32139 tablet.cc:2369] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:36:05.603866 32273 tablet.cc:2369] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:36:05.767190 32298 raft_consensus.cc:482] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:36:05.767643 32298 raft_consensus.cc:504] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbfd998697c64da28ea2d7ad50ed2770" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 39103 } } peers { permanent_uuid: "ca57816b5ffe45de8d2c9b3cd974fb04" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42081 } } peers { permanent_uuid: "642bb02c62e44cfaa3c4241856667cce" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 35795 } }
I20260923 17:36:05.769575 32298 leader_election.cc:288] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbfd998697c64da28ea2d7ad50ed2770 (127.20.240.66:39103), 642bb02c62e44cfaa3c4241856667cce (127.20.240.65:35795)
I20260923 17:36:05.775233 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:05.770105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:51605 (local address 127.20.240.66:39103)
0923 17:36:05.770314 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:05.770320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:05.770345 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:36:05.770426 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:05.770432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:05.770504 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:05.770620 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:36:05.771099 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.771872 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.773496 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.774311 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.774459 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:05.774537 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:05.774807 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 17:36:05.774942 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:36:05.775239 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:05.769850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:39103 (local address 127.20.240.67:51605)
0923 17:36:05.770074 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:05.770094 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:05.770257 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:05.770653 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:05.770662 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:05.770693 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:36:05.770948 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:05.770958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.772020 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.772024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:05.773322 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:05.773335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.774471 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.774477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:05.774650 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:05.774677 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:36:05.774759 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:36:05.774920 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 17:36:05.776211 32287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:05.770190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:58885 (local address 127.20.240.65:35795)
0923 17:36:05.770371 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:05.770377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:05.771031 (+ 654us) server_negotiation.cc:407] Connection header received
0923 17:36:05.771209 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:05.771216 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:05.771282 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:05.771360 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:36:05.771876 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.772555 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.774299 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.775115 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.775313 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:05.775389 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:05.775826 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 17:36:05.775968 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:36:05.776257 32299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:05.770053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:35795 (local address 127.20.240.67:58885)
0923 17:36:05.770920 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:05.770935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:05.771050 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:05.771389 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:05.771394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:05.771418 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:36:05.771720 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:05.771729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.772707 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.772713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:05.774115 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:05.774129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.775474 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.775482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:05.775658 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:05.775686 (+ 28us) client_negotiation.cc:769] Sending connection context
0923 17:36:05.775768 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:36:05.775918 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":536,"threads_started":1}
I20260923 17:36:05.776242 32093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74bf93fa907a41cd875b2139839f1563" candidate_uuid: "ca57816b5ffe45de8d2c9b3cd974fb04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbfd998697c64da28ea2d7ad50ed2770" is_pre_election: true
I20260923 17:36:05.776893 32093 raft_consensus.cc:2454] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca57816b5ffe45de8d2c9b3cd974fb04 in term 0.
I20260923 17:36:05.777354 31959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74bf93fa907a41cd875b2139839f1563" candidate_uuid: "ca57816b5ffe45de8d2c9b3cd974fb04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bb02c62e44cfaa3c4241856667cce" is_pre_election: true
I20260923 17:36:05.777565 32161 leader_election.cc:302] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca57816b5ffe45de8d2c9b3cd974fb04, fbfd998697c64da28ea2d7ad50ed2770; no voters:
I20260923 17:36:05.777881 31959 raft_consensus.cc:2454] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca57816b5ffe45de8d2c9b3cd974fb04 in term 0.
I20260923 17:36:05.777982 32298 raft_consensus.cc:2781] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:36:05.778100 32298 raft_consensus.cc:482] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:36:05.778192 32298 raft_consensus.cc:3037] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:36:05.780644 32298 raft_consensus.cc:504] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbfd998697c64da28ea2d7ad50ed2770" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 39103 } } peers { permanent_uuid: "ca57816b5ffe45de8d2c9b3cd974fb04" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42081 } } peers { permanent_uuid: "642bb02c62e44cfaa3c4241856667cce" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 35795 } }
I20260923 17:36:05.781358 32298 leader_election.cc:288] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [CANDIDATE]: Term 1 election: Requested vote from peers fbfd998697c64da28ea2d7ad50ed2770 (127.20.240.66:39103), 642bb02c62e44cfaa3c4241856667cce (127.20.240.65:35795)
I20260923 17:36:05.781919 32093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74bf93fa907a41cd875b2139839f1563" candidate_uuid: "ca57816b5ffe45de8d2c9b3cd974fb04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbfd998697c64da28ea2d7ad50ed2770"
I20260923 17:36:05.782060 31959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74bf93fa907a41cd875b2139839f1563" candidate_uuid: "ca57816b5ffe45de8d2c9b3cd974fb04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "642bb02c62e44cfaa3c4241856667cce"
I20260923 17:36:05.782168 32093 raft_consensus.cc:3037] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:36:05.782325 31959 raft_consensus.cc:3037] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:36:05.785532 32093 raft_consensus.cc:2454] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca57816b5ffe45de8d2c9b3cd974fb04 in term 1.
I20260923 17:36:05.785532 31959 raft_consensus.cc:2454] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca57816b5ffe45de8d2c9b3cd974fb04 in term 1.
I20260923 17:36:05.786139 32162 leader_election.cc:302] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 642bb02c62e44cfaa3c4241856667cce, ca57816b5ffe45de8d2c9b3cd974fb04; no voters:
I20260923 17:36:05.786509 32298 raft_consensus.cc:2781] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:36:05.786903 32298 raft_consensus.cc:686] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [term 1 LEADER]: Becoming Leader. State: Replica: ca57816b5ffe45de8d2c9b3cd974fb04, State: Running, Role: LEADER
I20260923 17:36:05.787429 32298 consensus_queue.cc:237] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbfd998697c64da28ea2d7ad50ed2770" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 39103 } } peers { permanent_uuid: "ca57816b5ffe45de8d2c9b3cd974fb04" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42081 } } peers { permanent_uuid: "642bb02c62e44cfaa3c4241856667cce" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 35795 } }
I20260923 17:36:05.792625 31814 catalog_manager.cc:5495] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca57816b5ffe45de8d2c9b3cd974fb04 (127.20.240.67). New cstate: current_term: 1 leader_uuid: "ca57816b5ffe45de8d2c9b3cd974fb04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbfd998697c64da28ea2d7ad50ed2770" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 39103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca57816b5ffe45de8d2c9b3cd974fb04" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 42081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "642bb02c62e44cfaa3c4241856667cce" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 35795 } health_report { overall_health: UNKNOWN } } }
I20260923 17:36:05.898859 32289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:05.893162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.20.240.67:42081)
0923 17:36:05.893362 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:05.893367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:05.893385 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:36:05.893692 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:05.893697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:05.893880 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:05.894015 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:36:05.894419 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.895252 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.896573 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.897041 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.897137 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:05.897654 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:36:05.898227 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:36:05.898342 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:05.898465 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 17:36:05.898537 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:36:05.907284 31959 raft_consensus.cc:1261] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce [term 1 FOLLOWER]: Refusing update from remote peer ca57816b5ffe45de8d2c9b3cd974fb04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:36:05.907282 32093 raft_consensus.cc:1261] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770 [term 1 FOLLOWER]: Refusing update from remote peer ca57816b5ffe45de8d2c9b3cd974fb04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:36:05.908360 32300 consensus_queue.cc:1035] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbfd998697c64da28ea2d7ad50ed2770" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 39103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:36:05.908758 32298 consensus_queue.cc:1035] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "642bb02c62e44cfaa3c4241856667cce" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 35795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:36:05.920461 32303 mvcc.cc:205] Tried to move back new op lower bound from 7332597620345511936 to 7332597619869208576. Current Snapshot: MvccSnapshot[applied={T|T < 7332597620345511936}]
I20260923 17:36:05.921432 32305 mvcc.cc:205] Tried to move back new op lower bound from 7332597620345511936 to 7332597619869208576. Current Snapshot: MvccSnapshot[applied={T|T < 7332597620345511936}]
I20260923 17:36:05.936100 31814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:36:05.936374 31814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:36:05.940194 31814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31db20be1d62413fada1b0c9e05a0a85: Sending DeleteTablet for 3 replicas of tablet 74bf93fa907a41cd875b2139839f1563
I20260923 17:36:05.941629 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet 74bf93fa907a41cd875b2139839f1563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:36:05 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:40272
I20260923 17:36:05.941684 32073 tablet_service.cc:1514] Processing DeleteTablet for tablet 74bf93fa907a41cd875b2139839f1563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:36:05 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:55296
I20260923 17:36:05.941653 31939 tablet_service.cc:1514] Processing DeleteTablet for tablet 74bf93fa907a41cd875b2139839f1563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:36:05 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:58658
I20260923 17:36:05.947348 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 31876
I20260923 17:36:05.954545 32313 tablet_replica.cc:330] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04: stopping tablet replica
I20260923 17:36:05.955678 32314 tablet_replica.cc:330] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770: stopping tablet replica
I20260923 17:36:05.956146 32314 raft_consensus.cc:2229] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:36:05.956732 32314 raft_consensus.cc:2258] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:36:05.955140 32313 raft_consensus.cc:2229] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:36:05.957844 32313 raft_consensus.cc:2258] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:36:05.959443 32315 tablet_replica.cc:330] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce: stopping tablet replica
I20260923 17:36:05.959926 32315 raft_consensus.cc:2229] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:36:05.960512 32315 raft_consensus.cc:2258] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:36:05.960588 32314 ts_tablet_manager.cc:1905] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:36:05.962993 32313 ts_tablet_manager.cc:1905] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:36:05.964390 32315 ts_tablet_manager.cc:1905] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:36:05.967834 32314 ts_tablet_manager.cc:1918] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:36:05.968005 32314 log.cc:1196] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/74bf93fa907a41cd875b2139839f1563
I20260923 17:36:05.968501 32314 ts_tablet_manager.cc:1939] T 74bf93fa907a41cd875b2139839f1563 P fbfd998697c64da28ea2d7ad50ed2770: Deleting consensus metadata
I20260923 17:36:05.970481 31800 catalog_manager.cc:4841] TS fbfd998697c64da28ea2d7ad50ed2770 (127.20.240.66:39103): tablet 74bf93fa907a41cd875b2139839f1563 (table test-table [id=7eaadcc70ee643ff9db65bcb0d5199f7]) successfully deleted
I20260923 17:36:05.970813 32313 ts_tablet_manager.cc:1918] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:36:05.971036 32313 log.cc:1196] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/74bf93fa907a41cd875b2139839f1563
I20260923 17:36:05.971485 32313 ts_tablet_manager.cc:1939] T 74bf93fa907a41cd875b2139839f1563 P ca57816b5ffe45de8d2c9b3cd974fb04: Deleting consensus metadata
I20260923 17:36:05.972992 31800 catalog_manager.cc:4841] TS ca57816b5ffe45de8d2c9b3cd974fb04 (127.20.240.67:42081): tablet 74bf93fa907a41cd875b2139839f1563 (table test-table [id=7eaadcc70ee643ff9db65bcb0d5199f7]) successfully deleted
I20260923 17:36:05.973613 32315 ts_tablet_manager.cc:1918] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:36:05.973798 32315 log.cc:1196] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/74bf93fa907a41cd875b2139839f1563
I20260923 17:36:05.974294 32315 ts_tablet_manager.cc:1939] T 74bf93fa907a41cd875b2139839f1563 P 642bb02c62e44cfaa3c4241856667cce: Deleting consensus metadata
I20260923 17:36:05.976248 31801 catalog_manager.cc:4841] TS 642bb02c62e44cfaa3c4241856667cce (127.20.240.65:35795): tablet 74bf93fa907a41cd875b2139839f1563 (table test-table [id=7eaadcc70ee643ff9db65bcb0d5199f7]) successfully deleted
Sep 23 17:36:05 dist-test-slave-ppjl krb5kdc[31770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:36:05.984957 32287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:05.948899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.20.240.65:35795)
0923 17:36:05.949089 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:05.949095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:05.960601 (+ 11506us) server_negotiation.cc:407] Connection header received
0923 17:36:05.961137 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:05.961144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:05.961239 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:05.961319 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:05.965786 (+ 4467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.966998 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.974649 (+ 7651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:05.975099 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:05.975290 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:05.979215 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:05.979230 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:05.979245 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:05.979283 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:05.982139 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:05.982852 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:05.982860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:05.982862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:05.982995 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:05.983427 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:05.983431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:05.983433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:05.983804 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:05.984158 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:05.984367 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 17:36:05.984580 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:36:05.985630 31999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:36:06.124332 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 31876
I20260923 17:36:06.140851 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 32010
Sep 23 17:36:06 dist-test-slave-ppjl krb5kdc[31770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:36:06.157469 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:06.142212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.20.240.66:39103)
0923 17:36:06.142409 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:06.142415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:06.142436 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:36:06.142502 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:06.142508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:06.142585 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:06.142703 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:06.144336 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:06.145041 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:06.146778 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:06.147206 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:06.147324 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:06.152032 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:06.152048 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:06.152064 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:06.152101 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:06.154760 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:06.155481 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:06.155489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:06.155491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:06.155610 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:06.155978 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:06.155981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:06.155983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:06.156373 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:06.156644 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:06.156935 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 17:36:06.157143 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:36:06.158272 32133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:36:06.248149 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 32010
I20260923 17:36:06.264328 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 32144
Sep 23 17:36:06 dist-test-slave-ppjl krb5kdc[31770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:36:06.281350 32289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:06.265230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.20.240.67:42081)
0923 17:36:06.265428 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:06.265437 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:06.265457 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:36:06.265721 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:06.265731 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:06.265818 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:06.265908 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:06.267220 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:06.268049 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:06.269302 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:06.269777 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:06.269895 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:06.274698 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:06.274716 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:06.274732 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:06.274780 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:06.278126 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:06.278903 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:06.278911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:06.278914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:06.279090 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:06.279650 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:06.279656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:06.279659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:06.280054 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:06.280354 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:06.280686 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 17:36:06.280940 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:36:06.282094 32267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:36:06.404398 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 32144
I20260923 17:36:06.419885 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 31783
I20260923 17:36:06.421125 31845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:36:06.529980 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 31783
2026-09-23T17:36:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4532 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:36:06 dist-test-slave-ppjl krb5kdc[32323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:36:06 dist-test-slave-ppjl krb5kdc[32323](info): set up 2 sockets
Sep 23 17:36:06 dist-test-slave-ppjl krb5kdc[32323](info): commencing operation
krb5kdc: starting...
W20260923 17:36:07.556985 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:36:07 dist-test-slave-ppjl krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:36:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:36:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/rangeradmin_127.20.240.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/rangeradmin_127.20.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/rangerlookup_127.20.240.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/rangerlookup_127.20.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/HTTP_127.20.240.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/HTTP_127.20.240.84@KRBTEST.COM.keytab.
I20260923 17:36:07.887068 21441 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSmAN3S/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/postgres -l logfile start
2026-09-23 17:36:10.500 UTC [32353] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 17:36:10.500 UTC [32353] LOG: listening on IPv4 address "127.20.240.84", port 34447
2026-09-23 17:36:10.504 UTC [32353] LOG: listening on Unix socket "/tmp/.s.PGSQL.34447"
2026-09-23 17:36:10.525 UTC [32356] LOG: database system was shut down at 2026-09-23 17:36:08 UTC
2026-09-23 17:36:10.528 UTC [32353] LOG: database system is ready to accept connections
I20260923 17:36:11.428155 21441 mini_postgres.cc:96] Postgres bound to 34447
2026-09-23 17:36:11.444 UTC [32364] FATAL: database "slave" does not exist
127.20.240.84:34447 - accepting connections
I20260923 17:36:11.445611 21441 mini_ranger.cc:161] Starting Ranger...
I20260923 17:36:11.471995 21441 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 17:36:11.994951 21441 mini_ranger.cc:87] Created ranger Postgres database
I20260923 17:36:11.995081 21441 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-admin
2026-09-23 17:36:12,289 [I] DB FLAVOR :POSTGRES
2026-09-23 17:36:12,290 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:36:12,290 [I] Checking connection..
2026-09-23 17:36:12,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 17:36:12,557 [I] Checking connection passed.
2026-09-23 17:36:12,557 [I] --------- Verifying version history table ---------
2026-09-23 17:36:12,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:36:12,793 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:36:12,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:36:13,031 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:36:13,032 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 17:36:13,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 17:36:13.266 UTC [32428] WARNING: there is no transaction in progress
2026-09-23 17:36:13,271 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 17:36:13,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:36:13,537 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 17:36:13,537 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:36:13,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 17:36:13,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 17:36:14,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ppjl.c.gcp-upstream.internal','N') ;"
2026-09-23 17:36:14,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:36:14,528 [I] Table x_portal_user does not exist in database ranger
2026-09-23 17:36:14,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:36:14,790 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 17:36:14,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:36:15,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:36:15,288 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 17:36:15,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 17:36:16.311 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:16.330 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:16.349 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:16.369 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:16.388 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:16.408 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:16.470 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:16.479 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:16.495 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:16.507 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:16.518 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:16.533 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:16.544 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:16.551 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:16.560 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:16.568 UTC [32536] WARNING: there is no transaction in progress
2026-09-23 17:36:17,353 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 17:36:17,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 17:36:17,578 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 17:36:17,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:36:17,814 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 17:36:17,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:36:18,046 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 17:36:18,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:36:18,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:36:18,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ppjl.c.gcp-upstream.internal';"
2026-09-23 17:36:18,800 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 17:36:18,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 17:36:19,047 [I] DB_PATCHES have already been applied
I20260923 17:36:19.053951 21441 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSmAN3S/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 17:36:19.054044 21441 mini_ranger.cc:193] Using host: 127.20.240.84
I20260923 17:36:19.068677 21441 mini_ranger.cc:227] Ranger admin URL: http://127.20.240.84:52873
Sep 23, 2026 5:36:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:36:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:36:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 5:36:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 5:36:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 5:36:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 5:36:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 17:36:20.107597 21441 mini_ranger.cc:160] Time spent starting Ranger: real 8.662s user 0.003s sys 0.039s
Sep 23, 2026 5:36:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 5:36:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 23, 2026 5:36:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 5:36:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 5:36:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 5:36:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 5:36:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 5:36:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 5:36:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 5:36:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 5:36:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 5:36:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 5:36:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 5:36:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 5:36:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 5:36:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 5:36:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 5:36:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 17:36:42.143874 21441 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/rangerkms_127.20.240.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/rangerkms_127.20.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.240.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.240.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.240.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/HTTP_127.20.240.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.240.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/HTTP_127.20.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 17:36:42 dist-test-slave-ppjl krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185002, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 17:36:42.315068 21441 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 17:36:42.339653 21441 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 17:36:43.307653 21441 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 17:36:43.307816 21441 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-kms
I20260923 17:36:43.307957 21441 mini_ranger_kms.cc:227] Using postgres at 127.20.240.84:34447
2026-09-23 17:36:43,722 [I] DB FLAVOR :POSTGRES
2026-09-23 17:36:43,722 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:36:43,722 [I] Checking connection
2026-09-23 17:36:43,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 17:36:43,979 [I] connection success
2026-09-23 17:36:43,979 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 17:36:43,979 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 17:36:43,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 17:36:44,242 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 17:36:44,242 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:36:44,243 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 17:36:44,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:34447/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 17:36:44,511 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 17:36:44.993899 21441 mini_ranger_kms.cc:310] Created kms service
I20260923 17:36:45.218676 21441 mini_ranger_kms.cc:326] Created kudu user
I20260923 17:36:45.268800 21441 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 17:36:45 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185002, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:36:45.658252 21441 mini_ranger_kms.cc:384] Added ranger policy
I20260923 17:36:45.658561 21441 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 17:36:45.658617 21441 mini_ranger_kms.cc:240] Using host: 127.20.240.84
I20260923 17:36:45.675302 21441 mini_ranger_kms.cc:276] Ranger KMS PID: 32716
I20260923 17:36:45.675527 21441 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.240.84:36409
Sep 23, 2026 5:36:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:36:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:36:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 5:36:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 5:36:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 5:36:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 17:36:47.793875 21441 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.479s user 0.116s sys 0.320s
I20260923 17:36:47.794144 21441 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 17:36:47.794425 21441 mini_ranger_kms.cc:401] 127.20.240.84:36409/kms/v1/keys
Sep 23 17:36:48 dist-test-slave-ppjl krb5kdc[32323](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790185008, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.240.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 17:36:48 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185008, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.240.84@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
Sep 23 17:36:48 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185008, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.240.84@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
Sep 23 17:36:51 dist-test-slave-ppjl krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:36:51.495699 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:45871
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:45871
--ranger_config_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.240.84:36409/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:36:51.901223 32767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:36:51.901649 32767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:36:51.901809 32767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:36:51.909934 32767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:36:51.910096 32767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:36:51.910189 32767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:36:51.910282 32767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:36:51.910368 32767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:36:51.921662 32767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.240.84:36409/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:45871
--ranger_config_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:45871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:36:51.923187 32767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:36:51.924578 32767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:36:51.937677 305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:51.937722 307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:51.937597 304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:36:51.938889 32767 server_base.cc:1034] running on GCE node
I20260923 17:36:51.939848 32767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:36:51.941365 32767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:36:51.942596 32767 hybrid_clock.cc:648] HybridClock initialized: now 1790185011942526 us; error 54 us; skew 500 ppm
I20260923 17:36:51.942984 32767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:36:51 dist-test-slave-ppjl krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185011, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:36:51.946766 32767 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:36:51.948482 32767 webserver.cc:460] Webserver started at http://127.20.240.126:44999/ using document root <none> and password file <none>
I20260923 17:36:51.949146 32767 fs_manager.cc:359] Metadata directory not provided
I20260923 17:36:51.949262 32767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:36:51.949548 32767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:36:51 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185011, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:36:52.006500 32767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "1cd6a872cba74107aafc24606c9993fa"
format_stamp: "Formatted at 2026-09-23 17:36:51 on dist-test-slave-ppjl"
server_key: "fe493a1b3b0520b30cb7bfaa20099eaf"
server_key_iv: "4dcf5bb2d366d8cef7e7bd7c8c3a83b3"
server_key_version: "kuduclusterkey@0"
I20260923 17:36:52.007843 32767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "1cd6a872cba74107aafc24606c9993fa"
format_stamp: "Formatted at 2026-09-23 17:36:51 on dist-test-slave-ppjl"
server_key: "fe493a1b3b0520b30cb7bfaa20099eaf"
server_key_iv: "4dcf5bb2d366d8cef7e7bd7c8c3a83b3"
server_key_version: "kuduclusterkey@0"
I20260923 17:36:52.015472 32767 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260923 17:36:52.038050 314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:52.039872 32767 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:36:52.040125 32767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "1cd6a872cba74107aafc24606c9993fa"
format_stamp: "Formatted at 2026-09-23 17:36:51 on dist-test-slave-ppjl"
server_key: "fe493a1b3b0520b30cb7bfaa20099eaf"
server_key_iv: "4dcf5bb2d366d8cef7e7bd7c8c3a83b3"
server_key_version: "kuduclusterkey@0"
I20260923 17:36:52.040424 32767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:36:52.082414 32767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:36:52.086956 32767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:36:52.087311 32767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:36:52.112326 32767 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:45871
I20260923 17:36:52.112341 365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:45871 every 8 connection(s)
I20260923 17:36:52.113919 32767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:36:52.116909 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 32767
I20260923 17:36:52.117091 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:36:52.117746 21441 external_mini_cluster.cc:1428] Setting key d4631031112f0a99269d95800a23b485
I20260923 17:36:52.118950 366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:36:52 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:36:52.136411 366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa: Bootstrap starting.
I20260923 17:36:52.141124 366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa: Neither blocks nor log segments found. Creating new log.
I20260923 17:36:52.142760 366 log.cc:824] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa: Log is configured to *not* fsync() on all Append() calls
I20260923 17:36:52.145118 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:52.119989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53290 (local address 127.20.240.126:45871)
0923 17:36:52.120882 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:52.120898 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:52.120958 (+ 60us) server_negotiation.cc:407] Connection header received
0923 17:36:52.122848 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:52.122878 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:52.123459 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:52.123974 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:52.125245 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:52.127509 (+ 2264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:52.128936 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:52.130030 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:52.130232 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:52.135174 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:52.135189 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:52.135213 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:52.135272 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:52.139856 (+ 4584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:52.140889 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:52.140898 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:52.140907 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:52.141123 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:52.141752 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:52.141757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:52.141761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:52.142588 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:52.142890 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:52.143415 (+ 525us) server_negotiation.cc:299] Negotiation successful
0923 17:36:52.144020 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":423,"threads_started":1}
I20260923 17:36:52.146306 366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa: No bootstrap required, opened a new log
I20260923 17:36:52.155597 366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd6a872cba74107aafc24606c9993fa" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45871 } }
I20260923 17:36:52.156000 366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:36:52.156107 366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd6a872cba74107aafc24606c9993fa, State: Initialized, Role: FOLLOWER
I20260923 17:36:52.156682 366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd6a872cba74107aafc24606c9993fa" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45871 } }
I20260923 17:36:52.156910 366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:36:52.157047 366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:36:52.157214 366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:36:52.160046 366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd6a872cba74107aafc24606c9993fa" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45871 } }
I20260923 17:36:52.160554 366 leader_election.cc:302] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cd6a872cba74107aafc24606c9993fa; no voters:
I20260923 17:36:52.161305 366 leader_election.cc:288] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:36:52.161441 371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:36:52.162438 371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [term 1 LEADER]: Becoming Leader. State: Replica: 1cd6a872cba74107aafc24606c9993fa, State: Running, Role: LEADER
I20260923 17:36:52.162992 371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd6a872cba74107aafc24606c9993fa" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45871 } }
I20260923 17:36:52.163621 366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:36:52.166127 372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cd6a872cba74107aafc24606c9993fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd6a872cba74107aafc24606c9993fa" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45871 } } }
I20260923 17:36:52.166222 373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cd6a872cba74107aafc24606c9993fa. Latest consensus state: current_term: 1 leader_uuid: "1cd6a872cba74107aafc24606c9993fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd6a872cba74107aafc24606c9993fa" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45871 } } }
I20260923 17:36:52.166620 372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [sys.catalog]: This master's current role is: LEADER
I20260923 17:36:52.166682 373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa [sys.catalog]: This master's current role is: LEADER
I20260923 17:36:52.167683 377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:36:52.174862 377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:36:52.177441 366 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 17:36:52.192970 377 catalog_manager.cc:1347] Generated new cluster ID: 07976fb1e96f486bbc4f269876d1e4ac
I20260923 17:36:52.193580 377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:36:52.214021 377 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:36:52.216503 377 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:36:52.228375 377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa: Generated new TSK 0
I20260923 17:36:52.229729 377 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:36:53.020018 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.240.84:36409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:45871
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:36:53 dist-test-slave-ppjl krb5kdc[32323](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790185013, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 17:36:53.506204 411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:36:53.506767 411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:36:53.507035 411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:36:53.518935 411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:36:53.519191 411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:36:53.519454 411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:36:53.534951 411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.240.84:36409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:45871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:36:53.536547 411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:36:53.538640 411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:36:53.562456 421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:53.568547 422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:53.579389 424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:36:53.584453 411 server_base.cc:1034] running on GCE node
I20260923 17:36:53.585386 411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:36:53.586815 411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:36:53.588115 411 hybrid_clock.cc:648] HybridClock initialized: now 1790185013587994 us; error 63 us; skew 500 ppm
I20260923 17:36:53.588599 411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:36:53 dist-test-slave-ppjl krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185013, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:36:53.596740 411 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:36:53.599776 411 webserver.cc:460] Webserver started at http://127.20.240.65:39917/ using document root <none> and password file <none>
I20260923 17:36:53.600762 411 fs_manager.cc:359] Metadata directory not provided
I20260923 17:36:53.600989 411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:36:53.601435 411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:36:53 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185013, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:36:53.623725 411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "c64b94c53ec2444895a8c681b3d71c86"
format_stamp: "Formatted at 2026-09-23 17:36:53 on dist-test-slave-ppjl"
server_key: "58e291801501f32452365ae22cad91d2"
server_key_iv: "0197882a4e4341757b3d77856722d190"
server_key_version: "kuduclusterkey@0"
I20260923 17:36:53.625391 411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "c64b94c53ec2444895a8c681b3d71c86"
format_stamp: "Formatted at 2026-09-23 17:36:53 on dist-test-slave-ppjl"
server_key: "58e291801501f32452365ae22cad91d2"
server_key_iv: "0197882a4e4341757b3d77856722d190"
server_key_version: "kuduclusterkey@0"
I20260923 17:36:53.635923 411 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.002s sys 0.009s
I20260923 17:36:53.666934 431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:53.669201 411 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.011s sys 0.000s
I20260923 17:36:53.669524 411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "c64b94c53ec2444895a8c681b3d71c86"
format_stamp: "Formatted at 2026-09-23 17:36:53 on dist-test-slave-ppjl"
server_key: "58e291801501f32452365ae22cad91d2"
server_key_iv: "0197882a4e4341757b3d77856722d190"
server_key_version: "kuduclusterkey@0"
I20260923 17:36:53.669932 411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:36:53.704479 411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:36:53.710851 411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:36:53.711668 411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:36:53.715122 411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:36:53.718176 411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:36:53.718397 411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260923 17:36:53.718640 411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:36:53.718760 411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:36:53 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185013, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:36:54.025658 411 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:40415
I20260923 17:36:54.025784 543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:40415 every 8 connection(s)
I20260923 17:36:54.027994 411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:36:54.036252 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 411
I20260923 17:36:54.036491 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:36:54.037336 21441 external_mini_cluster.cc:1428] Setting key 72c8bbaa3f2bd90e781c70c80687bbf8
Sep 23 17:36:54 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185013, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:36:54.060376 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:54.032302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:44383 (local address 127.20.240.126:45871)
0923 17:36:54.033063 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:54.033070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:54.035041 (+ 1971us) server_negotiation.cc:407] Connection header received
0923 17:36:54.039371 (+ 4330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:54.039376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:54.039511 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:54.039750 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:54.041968 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:54.042977 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:54.045368 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:54.046112 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:54.046235 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:54.051616 (+ 5381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:54.051625 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:54.051631 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:54.051682 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:54.055781 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:54.057083 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:54.057089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:54.057093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:54.057285 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:54.057952 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:54.057958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:54.057962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:54.058363 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:54.058524 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:54.059656 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0923 17:36:54.059912 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":476,"threads_started":1}
I20260923 17:36:54.061970 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:54.033048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:45871 (local address 127.20.240.65:44383)
0923 17:36:54.034692 (+ 1644us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:54.034751 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:54.038897 (+ 4146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:54.040113 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:54.040123 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:54.040829 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:36:54.041768 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:54.041792 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:54.043243 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:54.043248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:54.045206 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:54.045222 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:54.046452 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:54.046462 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:54.046911 (+ 449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:54.047799 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:36:54.047831 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:36:54.051228 (+ 3397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:36:54.056073 (+ 4845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:36:54.056085 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:36:54.056108 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:36:54.056885 (+ 777us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:36:54.057436 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:36:54.057441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:36:54.057444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:36:54.057804 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:36:54.058574 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:36:54.058584 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:36:54.059129 (+ 545us) client_negotiation.cc:769] Sending connection context
0923 17:36:54.059508 (+ 379us) client_negotiation.cc:240] Negotiation successful
0923 17:36:54.060197 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1173,"thread_start_us":776,"threads_started":1}
I20260923 17:36:54.064059 544 heartbeater.cc:347] Connected to a master server at 127.20.240.126:45871
I20260923 17:36:54.064455 544 heartbeater.cc:464] Registering TS with master...
I20260923 17:36:54.065532 544 heartbeater.cc:511] Master 127.20.240.126:45871 requested a full tablet report, sending...
I20260923 17:36:54.068588 331 ts_manager.cc:194] Registered new tserver with Master: c64b94c53ec2444895a8c681b3d71c86 (127.20.240.65:40415)
I20260923 17:36:54.071038 331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:44383
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:36:54.106910 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.240.84:36409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:45871
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:36:54 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185013, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
W20260923 17:36:54.769447 550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:36:54.769797 550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:36:54.769954 550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:36:54.778115 550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:36:54.778266 550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:36:54.778443 550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:36:54.789690 550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.240.84:36409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:45871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:36:54.791069 550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:36:54.792459 550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:36:54.807996 565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:54.808059 564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:54.808503 567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:36:54.809427 550 server_base.cc:1034] running on GCE node
I20260923 17:36:54.810070 550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:36:54.811127 550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:36:54.812520 550 hybrid_clock.cc:648] HybridClock initialized: now 1790185014812444 us; error 54 us; skew 500 ppm
I20260923 17:36:54.812891 550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:36:54 dist-test-slave-ppjl krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185014, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:36:54.816561 550 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:36:54.818389 550 webserver.cc:460] Webserver started at http://127.20.240.66:39881/ using document root <none> and password file <none>
I20260923 17:36:54.819149 550 fs_manager.cc:359] Metadata directory not provided
I20260923 17:36:54.819331 550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:36:54.819677 550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:36:54 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185014, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:36:54.840102 550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "7fa52f7c8ca147589c947c55f3973360"
format_stamp: "Formatted at 2026-09-23 17:36:54 on dist-test-slave-ppjl"
server_key: "994085dca15466f3b687a6f05c1e53e9"
server_key_iv: "ec71110e39e60006debbde9ef67d106f"
server_key_version: "kuduclusterkey@0"
I20260923 17:36:54.841317 550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "7fa52f7c8ca147589c947c55f3973360"
format_stamp: "Formatted at 2026-09-23 17:36:54 on dist-test-slave-ppjl"
server_key: "994085dca15466f3b687a6f05c1e53e9"
server_key_iv: "ec71110e39e60006debbde9ef67d106f"
server_key_version: "kuduclusterkey@0"
I20260923 17:36:54.849391 550 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.002s
I20260923 17:36:54.873752 574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:54.875866 550 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.005s sys 0.000s
I20260923 17:36:54.876076 550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "7fa52f7c8ca147589c947c55f3973360"
format_stamp: "Formatted at 2026-09-23 17:36:54 on dist-test-slave-ppjl"
server_key: "994085dca15466f3b687a6f05c1e53e9"
server_key_iv: "ec71110e39e60006debbde9ef67d106f"
server_key_version: "kuduclusterkey@0"
I20260923 17:36:54.876333 550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:36:54.901531 550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:36:54.906170 550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:36:54.906472 550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:36:54.907716 550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:36:54.909271 550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:36:54.909353 550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:54.909471 550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:36:54.909523 550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:54.962383 550 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:36633
I20260923 17:36:54.962512 686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:36633 every 8 connection(s)
I20260923 17:36:54.964102 550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:36:54.966154 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 550
I20260923 17:36:54.966320 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:36:54.966747 21441 external_mini_cluster.cc:1428] Setting key b36aaff68b7e4cd99cad8cda763479c3
Sep 23 17:36:54 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185014, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
I20260923 17:36:55.005441 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:54.967808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:34951 (local address 127.20.240.126:45871)
0923 17:36:54.983554 (+ 15746us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:54.983561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:54.984430 (+ 869us) server_negotiation.cc:407] Connection header received
0923 17:36:54.987128 (+ 2698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:54.987135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:54.987322 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:54.987478 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:54.989596 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:54.990448 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:54.992408 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:54.992848 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:54.992966 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:54.996937 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:54.996943 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:54.996946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:54.997004 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:55.000542 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:55.002209 (+ 1667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:55.002215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:55.002218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:55.002389 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:55.003231 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:55.003328 (+ 97us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:55.003333 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:55.003746 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:55.003937 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:55.004763 (+ 826us) server_negotiation.cc:299] Negotiation successful
0923 17:36:55.005017 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15587,"thread_start_us":15754,"threads_started":1}
I20260923 17:36:55.006588 690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:54.968600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:45871 (local address 127.20.240.66:34951)
0923 17:36:54.984023 (+ 15423us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:54.984087 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:54.986732 (+ 2645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:54.987860 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:54.987873 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:54.988619 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:36:54.989371 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:54.989395 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:54.990636 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:54.990641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:54.992228 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:54.992258 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:54.993122 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:54.993130 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:54.993411 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:54.994142 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:36:54.994176 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:36:54.996641 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:36:55.001250 (+ 4609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:36:55.001263 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:36:55.001277 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:36:55.001859 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:36:55.002645 (+ 786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:36:55.002651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:36:55.002653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:36:55.002951 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:36:55.003898 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:36:55.003916 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:36:55.004311 (+ 395us) client_negotiation.cc:769] Sending connection context
0923 17:36:55.004587 (+ 276us) client_negotiation.cc:240] Negotiation successful
0923 17:36:55.005201 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15067,"thread_start_us":14787,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
I20260923 17:36:55.008308 687 heartbeater.cc:347] Connected to a master server at 127.20.240.126:45871
I20260923 17:36:55.008639 687 heartbeater.cc:464] Registering TS with master...
I20260923 17:36:55.009371 687 heartbeater.cc:511] Master 127.20.240.126:45871 requested a full tablet report, sending...
I20260923 17:36:55.010948 331 ts_manager.cc:194] Registered new tserver with Master: 7fa52f7c8ca147589c947c55f3973360 (127.20.240.66:36633)
I20260923 17:36:55.011852 331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:34951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:36:55.043503 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.240.84:36409/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:45871
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:38999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:36:55.075377 544 heartbeater.cc:503] Master 127.20.240.126:45871 was elected leader, sending a full tablet report...
W20260923 17:36:55.461650 693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:36:55.461967 693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:36:55.462062 693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:36:55.470019 693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:36:55.470149 693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:36:55.470285 693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:36:55.481730 693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:38999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.240.84:36409/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:45871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:36:55.483100 693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:36:55.484488 693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:36:55.498696 698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:55.498939 699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:36:55.499173 701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:36:55.499879 693 server_base.cc:1034] running on GCE node
I20260923 17:36:55.500651 693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:36:55.501811 693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:36:55.503037 693 hybrid_clock.cc:648] HybridClock initialized: now 1790185015502973 us; error 46 us; skew 500 ppm
I20260923 17:36:55.503418 693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:36:55 dist-test-slave-ppjl krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185015, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:36:55.507040 693 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:36:55.508891 693 webserver.cc:460] Webserver started at http://127.20.240.67:39883/ using document root <none> and password file <none>
I20260923 17:36:55.509608 693 fs_manager.cc:359] Metadata directory not provided
I20260923 17:36:55.509720 693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:36:55.510007 693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:36:55 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185015, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:36:55.526314 693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "4e7f63b2cbf041b6aca2396d82550925"
format_stamp: "Formatted at 2026-09-23 17:36:55 on dist-test-slave-ppjl"
server_key: "63201da98d2acf266dc7c28b8a2699f8"
server_key_iv: "72e3bbe06440e13b3a4dfb5866654e49"
server_key_version: "kuduclusterkey@0"
I20260923 17:36:55.527261 693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "4e7f63b2cbf041b6aca2396d82550925"
format_stamp: "Formatted at 2026-09-23 17:36:55 on dist-test-slave-ppjl"
server_key: "63201da98d2acf266dc7c28b8a2699f8"
server_key_iv: "72e3bbe06440e13b3a4dfb5866654e49"
server_key_version: "kuduclusterkey@0"
I20260923 17:36:55.533704 693 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 17:36:55.552970 708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:55.554715 693 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260923 17:36:55.554908 693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "4e7f63b2cbf041b6aca2396d82550925"
format_stamp: "Formatted at 2026-09-23 17:36:55 on dist-test-slave-ppjl"
server_key: "63201da98d2acf266dc7c28b8a2699f8"
server_key_iv: "72e3bbe06440e13b3a4dfb5866654e49"
server_key_version: "kuduclusterkey@0"
I20260923 17:36:55.555138 693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:36:55.576365 693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:36:55.581156 693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:36:55.581471 693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:36:55.582636 693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:36:55.584331 693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:36:55.584462 693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:55.584594 693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:36:55.584648 693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:36:55.630169 693 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:41801
I20260923 17:36:55.630224 821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:41801 every 8 connection(s)
I20260923 17:36:55.631716 693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:36:55.632131 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 693
I20260923 17:36:55.632318 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:36:55.632916 21441 external_mini_cluster.cc:1428] Setting key 490a3783a700e50c47ede8a1a00cb3d2
Sep 23 17:36:55 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185015, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:36:55.655232 824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:55.634724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:36785 (local address 127.20.240.126:45871)
0923 17:36:55.635449 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:55.635455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:55.636609 (+ 1154us) server_negotiation.cc:407] Connection header received
0923 17:36:55.638690 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:55.638694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:55.638784 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:55.638910 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:55.640728 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.641617 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.643467 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.643977 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.644093 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:55.648276 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:55.648283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:55.648288 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:55.648388 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:55.651895 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:55.652733 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:55.652738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:55.652740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:55.652899 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:55.653398 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:55.653402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:55.653405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:55.653756 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:55.653875 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:55.654618 (+ 743us) server_negotiation.cc:299] Negotiation successful
0923 17:36:55.654821 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":807,"threads_started":1}
I20260923 17:36:55.656207 825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:55.635357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:45871 (local address 127.20.240.67:36785)
0923 17:36:55.636287 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:55.636347 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:55.638408 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:55.639133 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:55.639142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:55.639788 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:36:55.640546 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:55.640567 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.641776 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.641781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:55.643305 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:55.643318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.644187 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.644192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:55.644520 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:55.645227 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:36:55.645253 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:36:55.647963 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:36:55.652052 (+ 4089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:36:55.652060 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:36:55.652076 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:36:55.652612 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:36:55.652985 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:36:55.652988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:36:55.652990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:36:55.653191 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:36:55.653867 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:36:55.653873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:36:55.654220 (+ 347us) client_negotiation.cc:769] Sending connection context
0923 17:36:55.654462 (+ 242us) client_negotiation.cc:240] Negotiation successful
0923 17:36:55.654913 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":384,"threads_started":1}
I20260923 17:36:55.657941 822 heartbeater.cc:347] Connected to a master server at 127.20.240.126:45871
I20260923 17:36:55.658295 822 heartbeater.cc:464] Registering TS with master...
I20260923 17:36:55.659287 822 heartbeater.cc:511] Master 127.20.240.126:45871 requested a full tablet report, sending...
I20260923 17:36:55.661118 331 ts_manager.cc:194] Registered new tserver with Master: 4e7f63b2cbf041b6aca2396d82550925 (127.20.240.67:41801)
I20260923 17:36:55.662223 331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:36785
I20260923 17:36:55.663117 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:36:55.683941 824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:55.668599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53304 (local address 127.20.240.126:45871)
0923 17:36:55.668775 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:55.668779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:55.669443 (+ 664us) server_negotiation.cc:407] Connection header received
0923 17:36:55.669609 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:55.669612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:55.669703 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:55.669907 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:55.671737 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.672577 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.674172 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.674756 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.674879 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:55.677866 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:55.677874 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:55.677879 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:55.677918 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:55.680733 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:55.681652 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:55.681657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:55.681659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:55.681811 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:55.682315 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:55.682326 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:55.682330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:55.682752 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:55.682951 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:55.683436 (+ 485us) server_negotiation.cc:299] Negotiation successful
0923 17:36:55.683703 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:36:55.688551 331 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:36:55.696113 331 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:36:55.727386 836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:55.710933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34764 (local address 127.20.240.66:36633)
0923 17:36:55.712121 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:55.712128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:55.712155 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:36:55.712299 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:55.712307 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:55.712564 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:55.712832 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:36:55.715438 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.718819 (+ 3381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.723715 (+ 4896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.725582 (+ 1867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.725734 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:55.725954 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:55.726688 (+ 734us) server_negotiation.cc:299] Negotiation successful
0923 17:36:55.726886 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1052,"thread_start_us":920,"threads_started":1}
I20260923 17:36:55.728152 835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:55.710698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:36633 (local address 127.0.0.1:34764)
0923 17:36:55.711842 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:55.711883 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:55.712083 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:55.713190 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:55.713197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:55.713254 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:36:55.715191 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:55.715209 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.719017 (+ 3808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.719023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:55.723463 (+ 4440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:55.723484 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.726107 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.726114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:55.726358 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:55.726419 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 17:36:55.726567 (+ 148us) client_negotiation.cc:240] Negotiation successful
0923 17:36:55.726831 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"spinlock_wait_cycles":1920,"thread_start_us":711,"threads_started":1}
I20260923 17:36:55.733485 838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:55.712649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.20.240.65:40415)
0923 17:36:55.725437 (+ 12788us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:55.725446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:55.725477 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:36:55.725587 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:55.725597 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:55.725899 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:55.726168 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:36:55.727185 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.728994 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.731092 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.732260 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.732393 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:55.732669 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:55.732880 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 17:36:55.733051 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12576,"thread_start_us":12570,"threads_started":1}
I20260923 17:36:55.732810 622 tablet_service.cc:1467] Processing CreateTablet for tablet 28ec8b0b59af4b078dcf58942d983324 (DEFAULT_TABLE table=test-table [id=03f001d2343c4c50b20ca7adaf2b2932]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:36:55.734398 837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:55.712360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:40415 (local address 127.0.0.1:45490)
0923 17:36:55.713069 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:55.713088 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:55.713229 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:55.726157 (+ 12928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:55.726161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:55.726189 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:36:55.726892 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:55.726908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.729130 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.729136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:55.730922 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:55.730935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.732495 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.732501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:55.732678 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:55.732723 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 17:36:55.732833 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 17:36:55.732973 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":6,"thread_start_us":382,"threads_started":1}
I20260923 17:36:55.734557 622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28ec8b0b59af4b078dcf58942d983324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:36:55.735855 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:55.713870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.20.240.67:41801)
0923 17:36:55.726048 (+ 12178us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:55.726056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:55.726092 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:36:55.726184 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:55.726190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:55.726461 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:55.727587 (+ 1126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:36:55.727950 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.729524 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.732252 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.733825 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.733978 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:55.734220 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:55.735355 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0923 17:36:55.735536 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11988,"thread_start_us":11809,"threads_started":1}
I20260923 17:36:55.736421 839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:55.713108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:41801 (local address 127.0.0.1:56208)
0923 17:36:55.724943 (+ 11835us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:55.724966 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:55.725130 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:55.726755 (+ 1625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:55.726763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:55.726790 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:36:55.727393 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:55.727416 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.729701 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.729706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:55.732074 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:55.732089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.734227 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.734234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:55.734441 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:55.734474 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:36:55.735282 (+ 808us) client_negotiation.cc:240] Negotiation successful
0923 17:36:55.735420 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11698,"mutex_wait_us":43,"thread_start_us":11542,"threads_started":1}
I20260923 17:36:55.738615 479 tablet_service.cc:1467] Processing CreateTablet for tablet 28ec8b0b59af4b078dcf58942d983324 (DEFAULT_TABLE table=test-table [id=03f001d2343c4c50b20ca7adaf2b2932]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:36:55.739684 756 tablet_service.cc:1467] Processing CreateTablet for tablet 28ec8b0b59af4b078dcf58942d983324 (DEFAULT_TABLE table=test-table [id=03f001d2343c4c50b20ca7adaf2b2932]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:36:55.740509 479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28ec8b0b59af4b078dcf58942d983324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:36:55.741135 756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28ec8b0b59af4b078dcf58942d983324. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:36:55.752409 841 tablet_bootstrap.cc:492] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360: Bootstrap starting.
I20260923 17:36:55.754949 842 tablet_bootstrap.cc:492] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925: Bootstrap starting.
I20260923 17:36:55.757027 841 tablet_bootstrap.cc:654] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360: Neither blocks nor log segments found. Creating new log.
I20260923 17:36:55.758275 841 log.cc:824] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360: Log is configured to *not* fsync() on all Append() calls
I20260923 17:36:55.759495 843 tablet_bootstrap.cc:492] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86: Bootstrap starting.
I20260923 17:36:55.760039 842 tablet_bootstrap.cc:654] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925: Neither blocks nor log segments found. Creating new log.
I20260923 17:36:55.761646 842 log.cc:824] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925: Log is configured to *not* fsync() on all Append() calls
I20260923 17:36:55.764775 843 tablet_bootstrap.cc:654] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86: Neither blocks nor log segments found. Creating new log.
I20260923 17:36:55.766515 843 log.cc:824] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86: Log is configured to *not* fsync() on all Append() calls
I20260923 17:36:55.768934 842 tablet_bootstrap.cc:492] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925: No bootstrap required, opened a new log
I20260923 17:36:55.769007 841 tablet_bootstrap.cc:492] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360: No bootstrap required, opened a new log
I20260923 17:36:55.769264 842 ts_tablet_manager.cc:1397] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260923 17:36:55.769419 841 ts_tablet_manager.cc:1397] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20260923 17:36:55.779287 842 raft_consensus.cc:348] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa52f7c8ca147589c947c55f3973360" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36633 } } peers { permanent_uuid: "c64b94c53ec2444895a8c681b3d71c86" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40415 } } peers { permanent_uuid: "4e7f63b2cbf041b6aca2396d82550925" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41801 } }
I20260923 17:36:55.779846 842 raft_consensus.cc:374] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:36:55.780004 842 raft_consensus.cc:729] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7f63b2cbf041b6aca2396d82550925, State: Initialized, Role: FOLLOWER
I20260923 17:36:55.780117 843 tablet_bootstrap.cc:492] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86: No bootstrap required, opened a new log
I20260923 17:36:55.780551 843 ts_tablet_manager.cc:1397] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.011s
I20260923 17:36:55.780849 842 consensus_queue.cc:260] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa52f7c8ca147589c947c55f3973360" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36633 } } peers { permanent_uuid: "c64b94c53ec2444895a8c681b3d71c86" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40415 } } peers { permanent_uuid: "4e7f63b2cbf041b6aca2396d82550925" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41801 } }
I20260923 17:36:55.782184 822 heartbeater.cc:503] Master 127.20.240.126:45871 was elected leader, sending a full tablet report...
I20260923 17:36:55.782773 842 ts_tablet_manager.cc:1428] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260923 17:36:55.783447 841 raft_consensus.cc:348] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa52f7c8ca147589c947c55f3973360" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36633 } } peers { permanent_uuid: "c64b94c53ec2444895a8c681b3d71c86" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40415 } } peers { permanent_uuid: "4e7f63b2cbf041b6aca2396d82550925" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41801 } }
I20260923 17:36:55.783921 841 raft_consensus.cc:374] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:36:55.784047 841 raft_consensus.cc:729] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa52f7c8ca147589c947c55f3973360, State: Initialized, Role: FOLLOWER
I20260923 17:36:55.784663 841 consensus_queue.cc:260] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa52f7c8ca147589c947c55f3973360" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36633 } } peers { permanent_uuid: "c64b94c53ec2444895a8c681b3d71c86" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40415 } } peers { permanent_uuid: "4e7f63b2cbf041b6aca2396d82550925" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41801 } }
I20260923 17:36:55.785897 687 heartbeater.cc:503] Master 127.20.240.126:45871 was elected leader, sending a full tablet report...
I20260923 17:36:55.786648 841 ts_tablet_manager.cc:1428] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360: Time spent starting tablet: real 0.017s user 0.010s sys 0.005s
I20260923 17:36:55.792508 843 raft_consensus.cc:348] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa52f7c8ca147589c947c55f3973360" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36633 } } peers { permanent_uuid: "c64b94c53ec2444895a8c681b3d71c86" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40415 } } peers { permanent_uuid: "4e7f63b2cbf041b6aca2396d82550925" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41801 } }
I20260923 17:36:55.792999 843 raft_consensus.cc:374] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:36:55.793115 843 raft_consensus.cc:729] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c64b94c53ec2444895a8c681b3d71c86, State: Initialized, Role: FOLLOWER
I20260923 17:36:55.793777 843 consensus_queue.cc:260] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa52f7c8ca147589c947c55f3973360" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36633 } } peers { permanent_uuid: "c64b94c53ec2444895a8c681b3d71c86" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40415 } } peers { permanent_uuid: "4e7f63b2cbf041b6aca2396d82550925" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41801 } }
I20260923 17:36:55.796044 843 ts_tablet_manager.cc:1428] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260923 17:36:55.806114 848 raft_consensus.cc:482] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:36:55.806463 848 raft_consensus.cc:504] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa52f7c8ca147589c947c55f3973360" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36633 } } peers { permanent_uuid: "c64b94c53ec2444895a8c681b3d71c86" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40415 } } peers { permanent_uuid: "4e7f63b2cbf041b6aca2396d82550925" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41801 } }
I20260923 17:36:55.808295 848 leader_election.cc:288] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c64b94c53ec2444895a8c681b3d71c86 (127.20.240.65:40415), 4e7f63b2cbf041b6aca2396d82550925 (127.20.240.67:41801)
I20260923 17:36:55.816118 838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:55.808798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:38349 (local address 127.20.240.65:40415)
0923 17:36:55.809051 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:55.809059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:55.809740 (+ 681us) server_negotiation.cc:407] Connection header received
0923 17:36:55.809984 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:55.809990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:55.810074 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:55.810142 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:36:55.811147 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.812326 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.814233 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.815120 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.815325 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:55.815420 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:55.815706 (+ 286us) server_negotiation.cc:299] Negotiation successful
0923 17:36:55.815841 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:36:55.816495 850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:55.808702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:40415 (local address 127.20.240.66:38349)
0923 17:36:55.809554 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:55.809574 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:55.809780 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:55.810374 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:55.810379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:55.810412 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:36:55.811005 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:55.811017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.812469 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.812474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:55.814056 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:55.814070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.815392 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.815398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:55.815555 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:55.815579 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:36:55.815660 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:36:55.815807 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":1,"thread_start_us":493,"threads_started":1}
I20260923 17:36:55.817128 851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:55.809636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:41801 (local address 127.20.240.66:56593)
0923 17:36:55.810575 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0923 17:36:55.810607 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 17:36:55.810767 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:36:55.811268 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:36:55.811272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:36:55.811300 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:36:55.812016 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:55.812031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.813344 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.813347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:55.814862 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:36:55.814875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.816364 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.816373 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:36:55.816532 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:55.816557 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:36:55.816688 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 17:36:55.816829 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":577,"threads_started":1}
I20260923 17:36:55.817169 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:55.809959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:56593 (local address 127.20.240.67:41801)
0923 17:36:55.810171 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:55.810177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:55.810723 (+ 546us) server_negotiation.cc:407] Connection header received
0923 17:36:55.811027 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:55.811033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:55.811126 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:55.811256 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:36:55.812192 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.813176 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.815022 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.815986 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.816146 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:55.816218 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:55.816728 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 17:36:55.816874 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:36:55.817770 499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ec8b0b59af4b078dcf58942d983324" candidate_uuid: "7fa52f7c8ca147589c947c55f3973360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64b94c53ec2444895a8c681b3d71c86" is_pre_election: true
I20260923 17:36:55.818303 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ec8b0b59af4b078dcf58942d983324" candidate_uuid: "7fa52f7c8ca147589c947c55f3973360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7f63b2cbf041b6aca2396d82550925" is_pre_election: true
I20260923 17:36:55.818608 499 raft_consensus.cc:2454] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa52f7c8ca147589c947c55f3973360 in term 0.
I20260923 17:36:55.818899 777 raft_consensus.cc:2454] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa52f7c8ca147589c947c55f3973360 in term 0.
I20260923 17:36:55.819552 575 leader_election.cc:302] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa52f7c8ca147589c947c55f3973360, c64b94c53ec2444895a8c681b3d71c86; no voters:
I20260923 17:36:55.819968 848 raft_consensus.cc:2781] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:36:55.820148 848 raft_consensus.cc:482] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:36:55.820269 848 raft_consensus.cc:3037] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:36:55.823107 848 raft_consensus.cc:504] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa52f7c8ca147589c947c55f3973360" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36633 } } peers { permanent_uuid: "c64b94c53ec2444895a8c681b3d71c86" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40415 } } peers { permanent_uuid: "4e7f63b2cbf041b6aca2396d82550925" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41801 } }
I20260923 17:36:55.823827 848 leader_election.cc:288] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [CANDIDATE]: Term 1 election: Requested vote from peers c64b94c53ec2444895a8c681b3d71c86 (127.20.240.65:40415), 4e7f63b2cbf041b6aca2396d82550925 (127.20.240.67:41801)
I20260923 17:36:55.824555 777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ec8b0b59af4b078dcf58942d983324" candidate_uuid: "7fa52f7c8ca147589c947c55f3973360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7f63b2cbf041b6aca2396d82550925"
I20260923 17:36:55.824555 499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28ec8b0b59af4b078dcf58942d983324" candidate_uuid: "7fa52f7c8ca147589c947c55f3973360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c64b94c53ec2444895a8c681b3d71c86"
I20260923 17:36:55.824828 777 raft_consensus.cc:3037] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:36:55.824851 499 raft_consensus.cc:3037] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:36:55.828516 499 raft_consensus.cc:2454] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa52f7c8ca147589c947c55f3973360 in term 1.
I20260923 17:36:55.828634 777 raft_consensus.cc:2454] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa52f7c8ca147589c947c55f3973360 in term 1.
I20260923 17:36:55.829140 575 leader_election.cc:302] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fa52f7c8ca147589c947c55f3973360, c64b94c53ec2444895a8c681b3d71c86; no voters:
I20260923 17:36:55.829504 848 raft_consensus.cc:2781] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:36:55.830113 848 raft_consensus.cc:686] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa52f7c8ca147589c947c55f3973360, State: Running, Role: LEADER
I20260923 17:36:55.830639 848 consensus_queue.cc:237] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa52f7c8ca147589c947c55f3973360" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36633 } } peers { permanent_uuid: "c64b94c53ec2444895a8c681b3d71c86" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40415 } } peers { permanent_uuid: "4e7f63b2cbf041b6aca2396d82550925" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41801 } }
I20260923 17:36:55.836063 330 catalog_manager.cc:5495] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa52f7c8ca147589c947c55f3973360 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "7fa52f7c8ca147589c947c55f3973360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa52f7c8ca147589c947c55f3973360" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c64b94c53ec2444895a8c681b3d71c86" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e7f63b2cbf041b6aca2396d82550925" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41801 } health_report { overall_health: UNKNOWN } } }
I20260923 17:36:55.881340 836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:55.874931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.20.240.66:36633)
0923 17:36:55.875209 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:55.875215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:55.875237 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:36:55.875452 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:55.875456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:55.875538 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:55.875672 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:36:55.876115 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.877034 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.878862 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.879405 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.879524 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:55.880161 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:36:55.880693 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:36:55.880857 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:55.880940 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 17:36:55.881027 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260923 17:36:55.883595 823 tablet.cc:2369] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:36:55.890667 777 raft_consensus.cc:1261] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925 [term 1 FOLLOWER]: Refusing update from remote peer 7fa52f7c8ca147589c947c55f3973360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:36:55.890730 499 raft_consensus.cc:1261] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86 [term 1 FOLLOWER]: Refusing update from remote peer 7fa52f7c8ca147589c947c55f3973360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:36:55.891682 852 consensus_queue.cc:1035] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e7f63b2cbf041b6aca2396d82550925" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:36:55.892053 848 consensus_queue.cc:1035] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c64b94c53ec2444895a8c681b3d71c86" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:36:55.921998 330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:36:55.922362 330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:36:55.926936 330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cd6a872cba74107aafc24606c9993fa: Sending DeleteTablet for 3 replicas of tablet 28ec8b0b59af4b078dcf58942d983324
I20260923 17:36:55.928633 622 tablet_service.cc:1514] Processing DeleteTablet for tablet 28ec8b0b59af4b078dcf58942d983324 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:36:55 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:34764
I20260923 17:36:55.928648 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 28ec8b0b59af4b078dcf58942d983324 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:36:55 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:56208
I20260923 17:36:55.930004 866 tablet_replica.cc:330] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925: stopping tablet replica
I20260923 17:36:55.930222 865 tablet_replica.cc:330] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360: stopping tablet replica
I20260923 17:36:55.930228 479 tablet_service.cc:1514] Processing DeleteTablet for tablet 28ec8b0b59af4b078dcf58942d983324 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:36:55 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:45490
I20260923 17:36:55.930476 866 raft_consensus.cc:2229] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:36:55.930630 865 raft_consensus.cc:2229] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:36:55.930996 866 raft_consensus.cc:2258] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:36:55.931137 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 411
I20260923 17:36:55.931394 865 raft_consensus.cc:2258] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:36:55.934787 865 ts_tablet_manager.cc:1905] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:36:55.942198 867 tablet_replica.cc:330] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86: stopping tablet replica
I20260923 17:36:55.942858 867 raft_consensus.cc:2229] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:36:55.943531 867 raft_consensus.cc:2258] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:36:55.944370 865 ts_tablet_manager.cc:1918] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:36:55.944733 865 log.cc:1196] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/28ec8b0b59af4b078dcf58942d983324
I20260923 17:36:55.945363 865 ts_tablet_manager.cc:1939] T 28ec8b0b59af4b078dcf58942d983324 P 7fa52f7c8ca147589c947c55f3973360: Deleting consensus metadata
I20260923 17:36:55.945622 866 ts_tablet_manager.cc:1905] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:36:55.946039 867 ts_tablet_manager.cc:1905] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:36:55.947477 315 catalog_manager.cc:4841] TS 7fa52f7c8ca147589c947c55f3973360 (127.20.240.66:36633): tablet 28ec8b0b59af4b078dcf58942d983324 (table test-table [id=03f001d2343c4c50b20ca7adaf2b2932]) successfully deleted
I20260923 17:36:55.954048 866 ts_tablet_manager.cc:1918] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:36:55.954101 867 ts_tablet_manager.cc:1918] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:36:55.954317 867 log.cc:1196] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/28ec8b0b59af4b078dcf58942d983324
I20260923 17:36:55.954458 866 log.cc:1196] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/28ec8b0b59af4b078dcf58942d983324
I20260923 17:36:55.955178 867 ts_tablet_manager.cc:1939] T 28ec8b0b59af4b078dcf58942d983324 P c64b94c53ec2444895a8c681b3d71c86: Deleting consensus metadata
I20260923 17:36:55.955190 866 ts_tablet_manager.cc:1939] T 28ec8b0b59af4b078dcf58942d983324 P 4e7f63b2cbf041b6aca2396d82550925: Deleting consensus metadata
I20260923 17:36:55.957472 315 catalog_manager.cc:4841] TS c64b94c53ec2444895a8c681b3d71c86 (127.20.240.65:40415): tablet 28ec8b0b59af4b078dcf58942d983324 (table test-table [id=03f001d2343c4c50b20ca7adaf2b2932]) successfully deleted
I20260923 17:36:55.957842 315 catalog_manager.cc:4841] TS 4e7f63b2cbf041b6aca2396d82550925 (127.20.240.67:41801): tablet 28ec8b0b59af4b078dcf58942d983324 (table test-table [id=03f001d2343c4c50b20ca7adaf2b2932]) successfully deleted
Sep 23 17:36:55 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:36:55.971542 838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:55.932238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45494 (local address 127.20.240.65:40415)
0923 17:36:55.932427 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:55.932433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:55.951489 (+ 19056us) server_negotiation.cc:407] Connection header received
0923 17:36:55.952039 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:55.952045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:55.952129 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:55.952205 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:55.954593 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.955853 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.957924 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:55.958746 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:55.958871 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:55.963811 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:55.963834 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:55.963851 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:55.963905 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:55.967848 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:55.968784 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:55.968792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:55.968795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:55.968994 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:55.969433 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:55.969440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:55.969443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:55.969928 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:55.970266 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:55.970671 (+ 405us) server_negotiation.cc:299] Negotiation successful
0923 17:36:55.970917 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:36:55.972254 539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:36:56.072854 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 411
I20260923 17:36:56.087757 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 550
Sep 23 17:36:56 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:36:56.106146 836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:56.088623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34780 (local address 127.20.240.66:36633)
0923 17:36:56.088815 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:56.088820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:56.088931 (+ 111us) server_negotiation.cc:407] Connection header received
0923 17:36:56.089132 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:56.089136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:56.089216 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:56.089310 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:56.091134 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:56.092053 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:56.094175 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:56.094667 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:56.094814 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:56.100128 (+ 5314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:56.100145 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:56.100161 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:56.100221 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:56.103273 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:56.104113 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:56.104120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:56.104122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:56.104275 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:56.104696 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:56.104699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:56.104701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:56.105060 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:56.105294 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:56.105665 (+ 371us) server_negotiation.cc:299] Negotiation successful
0923 17:36:56.105851 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:36:56.106850 682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:36:56.221982 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 550
I20260923 17:36:56.236800 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 693
Sep 23 17:36:56 dist-test-slave-ppjl krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:36:56.253064 840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:36:56.237575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.20.240.67:41801)
0923 17:36:56.237747 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 17:36:56.237752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:36:56.237769 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:36:56.237949 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:36:56.237952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:36:56.238013 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:36:56.238092 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:36:56.239431 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:56.240165 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:56.241476 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:36:56.241878 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:36:56.241976 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:36:56.246505 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:36:56.246523 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:36:56.246539 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:36:56.246585 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:36:56.249472 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:56.250276 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:56.250283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:56.250286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:56.250465 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:36:56.251001 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:36:56.251005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:36:56.251007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:36:56.251779 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:36:56.252048 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0923 17:36:56.252488 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 17:36:56.252678 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:36:56.253827 817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:36:56.342723 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 693
I20260923 17:36:56.356086 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 32767
I20260923 17:36:56.357411 361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:36:56.471437 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 32767
I20260923 17:36:56.494053 21441 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 17:36:56.829948 21441 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 17:36:56.830159 21441 mini_ranger.cc:66] Stopping Ranger...
I20260923 17:36:57.176265 21441 mini_ranger.cc:68] Stopped Ranger
2026-09-23 17:36:57.176 UTC [32353] LOG: received smart shutdown request
2026-09-23 17:36:57.180 UTC [32353] LOG: background worker "logical replication launcher" (PID 32362) exited with exit code 1
2026-09-23 17:36:57.180 UTC [32357] LOG: shutting down
2026-09-23 17:36:57.380 UTC [32353] LOG: database system is shut down
2026-09-23T17:36:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50960 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:36:57 dist-test-slave-ppjl krb5kdc[881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:36:57 dist-test-slave-ppjl krb5kdc[881](info): set up 2 sockets
Sep 23 17:36:57 dist-test-slave-ppjl krb5kdc[881](info): commencing operation
krb5kdc: starting...
W20260923 17:36:58.515997 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:36:58 dist-test-slave-ppjl krb5kdc[881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:36:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:36:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/rangeradmin_127.20.240.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/rangeradmin_127.20.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/rangerlookup_127.20.240.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/rangerlookup_127.20.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/HTTP_127.20.240.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/HTTP_127.20.240.84@KRBTEST.COM.keytab.
I20260923 17:36:58.846879 21441 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSmAN3S/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/postgres -l logfile start
2026-09-23 17:37:01.350 UTC [911] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 17:37:01.350 UTC [911] LOG: listening on IPv4 address "127.20.240.84", port 36727
2026-09-23 17:37:01.354 UTC [911] LOG: listening on Unix socket "/tmp/.s.PGSQL.36727"
2026-09-23 17:37:01.375 UTC [914] LOG: database system was shut down at 2026-09-23 17:36:59 UTC
2026-09-23 17:37:01.380 UTC [911] LOG: database system is ready to accept connections
I20260923 17:37:02.267688 21441 mini_postgres.cc:96] Postgres bound to 36727
2026-09-23 17:37:02.283 UTC [922] FATAL: database "slave" does not exist
127.20.240.84:36727 - accepting connections
I20260923 17:37:02.284403 21441 mini_ranger.cc:161] Starting Ranger...
I20260923 17:37:02.309507 21441 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 17:37:02.815806 21441 mini_ranger.cc:87] Created ranger Postgres database
I20260923 17:37:02.815917 21441 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-admin
2026-09-23 17:37:03,101 [I] DB FLAVOR :POSTGRES
2026-09-23 17:37:03,102 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:37:03,102 [I] Checking connection..
2026-09-23 17:37:03,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 17:37:03,368 [I] Checking connection passed.
2026-09-23 17:37:03,368 [I] --------- Verifying version history table ---------
2026-09-23 17:37:03,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:37:03,616 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:37:03,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:37:03,873 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:37:03,873 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 17:37:03,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 17:37:04.118 UTC [986] WARNING: there is no transaction in progress
2026-09-23 17:37:04,124 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 17:37:04,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:37:04,358 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 17:37:04,359 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:37:04,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 17:37:04,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 17:37:04,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ppjl.c.gcp-upstream.internal','N') ;"
2026-09-23 17:37:05,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:37:05,352 [I] Table x_portal_user does not exist in database ranger
2026-09-23 17:37:05,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:37:05,606 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 17:37:05,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:37:05,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:37:06,085 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 17:37:06,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 17:37:07.128 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:07.145 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:07.164 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:07.182 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:07.200 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:07.218 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:07.281 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:07.290 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:07.307 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:07.319 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:07.331 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:07.347 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:07.358 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:07.366 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:07.373 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:07.381 UTC [1094] WARNING: there is no transaction in progress
2026-09-23 17:37:08,153 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 17:37:08,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 17:37:08,378 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 17:37:08,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:37:08,660 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 17:37:08,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:37:08,926 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 17:37:08,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:37:09,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:37:09,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ppjl.c.gcp-upstream.internal';"
2026-09-23 17:37:09,680 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 17:37:09,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 17:37:09,944 [I] DB_PATCHES have already been applied
I20260923 17:37:09.950685 21441 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSmAN3S/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 17:37:09.950778 21441 mini_ranger.cc:193] Using host: 127.20.240.84
I20260923 17:37:09.964728 21441 mini_ranger.cc:227] Ranger admin URL: http://127.20.240.84:58251
Sep 23, 2026 5:37:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:37:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:37:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 5:37:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 5:37:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 5:37:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 5:37:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 17:37:11.013415 21441 mini_ranger.cc:160] Time spent starting Ranger: real 8.729s user 0.004s sys 0.036s
Sep 23, 2026 5:37:31 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 5:37:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 23, 2026 5:37:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 5:37:31 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 5:37:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 5:37:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 5:37:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 5:37:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 5:37:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 5:37:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 5:37:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 5:37:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 5:37:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 5:37:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 5:37:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 5:37:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 5:37:31 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 5:37:32 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 17:37:33.539752 21441 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/rangerkms_127.20.240.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/rangerkms_127.20.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.240.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.240.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.240.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/HTTP_127.20.240.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.240.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/HTTP_127.20.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 17:37:33 dist-test-slave-ppjl krb5kdc[881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185053, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 17:37:33.695150 21441 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 17:37:33.714867 21441 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 17:37:34.613691 21441 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 17:37:34.613794 21441 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-kms
I20260923 17:37:34.613837 21441 mini_ranger_kms.cc:227] Using postgres at 127.20.240.84:36727
2026-09-23 17:37:34,949 [I] DB FLAVOR :POSTGRES
2026-09-23 17:37:34,950 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:37:34,950 [I] Checking connection
2026-09-23 17:37:34,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 17:37:35,191 [I] connection success
2026-09-23 17:37:35,191 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 17:37:35,192 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 17:37:35,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 17:37:35,417 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 17:37:35,417 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:37:35,418 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 17:37:35,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.240.84:36727/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 17:37:35,681 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 17:37:36.184937 21441 mini_ranger_kms.cc:310] Created kms service
I20260923 17:37:36.358265 21441 mini_ranger_kms.cc:326] Created kudu user
I20260923 17:37:36.408190 21441 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 17:37:36 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185053, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:37:36.812412 21441 mini_ranger_kms.cc:384] Added ranger policy
I20260923 17:37:36.812718 21441 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSmAN3S/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSmAN3S/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 17:37:36.812757 21441 mini_ranger_kms.cc:240] Using host: 127.20.240.84
I20260923 17:37:36.826783 21441 mini_ranger_kms.cc:276] Ranger KMS PID: 1276
I20260923 17:37:36.826996 21441 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.240.84:40793
Sep 23, 2026 5:37:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:37:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:37:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 5:37:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 5:37:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 5:37:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 17:37:38.954088 21441 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.259s user 0.096s sys 0.253s
I20260923 17:37:38.954308 21441 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 17:37:38.954463 21441 mini_ranger_kms.cc:401] 127.20.240.84:40793/kms/v1/keys
Sep 23 17:37:39 dist-test-slave-ppjl krb5kdc[881](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790185059, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.240.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 17:37:39 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185059, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.240.84@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
Sep 23 17:37:39 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185059, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.240.84@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
Sep 23 17:37:42 dist-test-slave-ppjl krb5kdc[881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:37:42.183194 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:45003
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:45003
--ranger_config_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.240.84:40793/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:37:42.519125 1326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:37:42.519495 1326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:37:42.519641 1326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:37:42.527052 1326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:37:42.527202 1326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:37:42.527281 1326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:37:42.527354 1326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:37:42.527418 1326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:37:42.537626 1326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.240.84:40793/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:45003
--ranger_config_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:45003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:37:42.539027 1326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:37:42.540608 1326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:37:42.552975 1331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:42.552800 1332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:42.552919 1334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:37:42.553363 1326 server_base.cc:1034] running on GCE node
I20260923 17:37:42.554363 1326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:37:42.555939 1326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:37:42.557199 1326 hybrid_clock.cc:648] HybridClock initialized: now 1790185062557153 us; error 39 us; skew 500 ppm
I20260923 17:37:42.557601 1326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:37:42 dist-test-slave-ppjl krb5kdc[881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185062, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:37:42.561054 1326 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:37:42.562695 1326 webserver.cc:460] Webserver started at http://127.20.240.126:46853/ using document root <none> and password file <none>
I20260923 17:37:42.563450 1326 fs_manager.cc:359] Metadata directory not provided
I20260923 17:37:42.563575 1326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:37:42.563901 1326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:37:42 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185062, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:37:42.628810 1326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "3a455f6cb1df416db910571ea726ab17"
format_stamp: "Formatted at 2026-09-23 17:37:42 on dist-test-slave-ppjl"
server_key: "1a2efb0019c787c10e4666bc23a443c2"
server_key_iv: "a7a4bb1f82059beae69fca1a8a1ac34b"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:42.629664 1326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "3a455f6cb1df416db910571ea726ab17"
format_stamp: "Formatted at 2026-09-23 17:37:42 on dist-test-slave-ppjl"
server_key: "1a2efb0019c787c10e4666bc23a443c2"
server_key_iv: "a7a4bb1f82059beae69fca1a8a1ac34b"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:42.635322 1326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:37:42.653378 1341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:42.655367 1326 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:37:42.655627 1326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "3a455f6cb1df416db910571ea726ab17"
format_stamp: "Formatted at 2026-09-23 17:37:42 on dist-test-slave-ppjl"
server_key: "1a2efb0019c787c10e4666bc23a443c2"
server_key_iv: "a7a4bb1f82059beae69fca1a8a1ac34b"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:42.655925 1326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:37:42.678697 1326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:37:42.683493 1326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:37:42.683801 1326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:37:42.702742 1326 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:45003
I20260923 17:37:42.702746 1392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:45003 every 8 connection(s)
I20260923 17:37:42.704154 1326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:37:42.708027 1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:37:42.708658 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 1326
I20260923 17:37:42.708846 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:37:42.709527 21441 external_mini_cluster.cc:1428] Setting key 3004d12a33edadeb246c4c96098e69e8
I20260923 17:37:42.722149 1393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17: Bootstrap starting.
Sep 23 17:37:42 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:37:42.725404 1393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17: Neither blocks nor log segments found. Creating new log.
I20260923 17:37:42.726712 1393 log.cc:824] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17: Log is configured to *not* fsync() on all Append() calls
I20260923 17:37:42.730526 1393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17: No bootstrap required, opened a new log
I20260923 17:37:42.733789 1396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:42.711719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.20.240.126:45003)
0923 17:37:42.712576 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:42.712592 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:42.712647 (+ 55us) server_negotiation.cc:407] Connection header received
0923 17:37:42.714482 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:42.714514 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:42.715075 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:42.715577 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:42.716742 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:42.718706 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:42.720120 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:42.721146 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:42.721318 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:42.725162 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:42.725174 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:42.725193 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:42.725250 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:42.728956 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:42.729920 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:42.729928 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:42.729937 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:42.730113 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:42.730631 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:42.730637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:42.730640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:42.731308 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:42.731554 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:42.732136 (+ 582us) server_negotiation.cc:299] Negotiation successful
0923 17:37:42.732531 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":377,"threads_started":1}
I20260923 17:37:42.739959 1393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a455f6cb1df416db910571ea726ab17" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45003 } }
I20260923 17:37:42.740412 1393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:37:42.740525 1393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a455f6cb1df416db910571ea726ab17, State: Initialized, Role: FOLLOWER
I20260923 17:37:42.741102 1393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a455f6cb1df416db910571ea726ab17" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45003 } }
I20260923 17:37:42.741315 1393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:37:42.741415 1393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:37:42.741540 1393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:37:42.744203 1393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a455f6cb1df416db910571ea726ab17" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45003 } }
I20260923 17:37:42.744685 1393 leader_election.cc:302] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a455f6cb1df416db910571ea726ab17; no voters:
I20260923 17:37:42.745246 1393 leader_election.cc:288] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:37:42.745337 1398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:37:42.746528 1398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 1 LEADER]: Becoming Leader. State: Replica: 3a455f6cb1df416db910571ea726ab17, State: Running, Role: LEADER
I20260923 17:37:42.747059 1398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a455f6cb1df416db910571ea726ab17" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45003 } }
I20260923 17:37:42.748024 1393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:37:42.750146 1399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a455f6cb1df416db910571ea726ab17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a455f6cb1df416db910571ea726ab17" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45003 } } }
I20260923 17:37:42.750517 1399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [sys.catalog]: This master's current role is: LEADER
I20260923 17:37:42.750963 1400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a455f6cb1df416db910571ea726ab17. Latest consensus state: current_term: 1 leader_uuid: "3a455f6cb1df416db910571ea726ab17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a455f6cb1df416db910571ea726ab17" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45003 } } }
I20260923 17:37:42.751330 1400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [sys.catalog]: This master's current role is: LEADER
I20260923 17:37:42.751477 1404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:37:42.758675 1404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:37:42.768994 1393 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 17:37:42.769445 1404 catalog_manager.cc:1347] Generated new cluster ID: 66e46c9969284e02ab8904201b3a5044
I20260923 17:37:42.769563 1404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:37:42.821203 1404 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:37:42.823423 1404 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:37:42.843715 1404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17: Generated new TSK 0
I20260923 17:37:42.844864 1404 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:37:43.648523 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.240.84:40793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:45003
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:37:43 dist-test-slave-ppjl krb5kdc[881](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790185063, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 17:37:44.130349 1438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:37:44.130870 1438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:37:44.131075 1438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:37:44.142755 1438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:37:44.142993 1438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:37:44.143244 1438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:37:44.159824 1438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.240.84:40793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:45003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:37:44.161792 1438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:37:44.163743 1438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:37:44.182556 1448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:44.182578 1449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:44.192308 1451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:37:44.193415 1438 server_base.cc:1034] running on GCE node
I20260923 17:37:44.194110 1438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:37:44.201330 1438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:37:44.204906 1438 hybrid_clock.cc:648] HybridClock initialized: now 1790185064204806 us; error 73 us; skew 500 ppm
I20260923 17:37:44.205410 1438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:37:44 dist-test-slave-ppjl krb5kdc[881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185064, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:37:44.214838 1438 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:37:44.222841 1438 webserver.cc:460] Webserver started at http://127.20.240.65:37475/ using document root <none> and password file <none>
I20260923 17:37:44.223752 1438 fs_manager.cc:359] Metadata directory not provided
I20260923 17:37:44.223892 1438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:37:44.224406 1438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:37:44 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185064, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:37:44.243816 1438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "e0e1f02b5e7344e1af63a9981b64c619"
format_stamp: "Formatted at 2026-09-23 17:37:44 on dist-test-slave-ppjl"
server_key: "168b77152897537ccb7a692a1d867e43"
server_key_iv: "4b9018218aa96b78efeb79e83a2f340b"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:44.244704 1438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "e0e1f02b5e7344e1af63a9981b64c619"
format_stamp: "Formatted at 2026-09-23 17:37:44 on dist-test-slave-ppjl"
server_key: "168b77152897537ccb7a692a1d867e43"
server_key_iv: "4b9018218aa96b78efeb79e83a2f340b"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:44.250473 1438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:37:44.284587 1458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:44.286794 1438 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.006s sys 0.004s
I20260923 17:37:44.287061 1438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "e0e1f02b5e7344e1af63a9981b64c619"
format_stamp: "Formatted at 2026-09-23 17:37:44 on dist-test-slave-ppjl"
server_key: "168b77152897537ccb7a692a1d867e43"
server_key_iv: "4b9018218aa96b78efeb79e83a2f340b"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:44.287376 1438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:37:44.340612 1438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:37:44.346338 1438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:37:44.346678 1438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:37:44.347968 1438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:37:44.354508 1438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:37:44.354619 1438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:44.354737 1438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:37:44.354784 1438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:44.424794 1438 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:46453
I20260923 17:37:44.424832 1570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:46453 every 8 connection(s)
I20260923 17:37:44.427183 1438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:37:44.437395 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 1438
I20260923 17:37:44.437611 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:37:44.438452 21441 external_mini_cluster.cc:1428] Setting key 3ca15d3f02bd7956e150430037ac5469
Sep 23 17:37:44 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185063, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
Sep 23 17:37:44 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185064, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:37:44.512503 1574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:44.458362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:57821 (local address 127.20.240.126:45003)
0923 17:37:44.470657 (+ 12295us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:44.470665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:44.477516 (+ 6851us) server_negotiation.cc:407] Connection header received
0923 17:37:44.483196 (+ 5680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:44.483203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:44.483309 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:44.483397 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:44.485954 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:44.487193 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:44.489780 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:44.490513 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:44.490674 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:44.504137 (+ 13463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:44.504146 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:44.504152 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:44.504216 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:44.508128 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:44.509313 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:44.509320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:44.509323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:44.509508 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:44.510180 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:44.510186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:44.510189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:44.510575 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:44.510695 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:44.511854 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0923 17:37:44.512107 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12150,"thread_start_us":11996,"threads_started":1}
I20260923 17:37:44.514865 1575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:44.458860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:45003 (local address 127.20.240.65:57821)
0923 17:37:44.477269 (+ 18409us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:44.477340 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:44.482689 (+ 5349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:44.484050 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:44.484063 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:44.484791 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:44.485685 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:44.485714 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:44.487443 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:44.487449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:44.489548 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:44.489566 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:44.491050 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:44.491059 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:44.491462 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:44.492269 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:44.492305 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:44.503760 (+ 11455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:44.508375 (+ 4615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:44.508386 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:44.508407 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:44.509138 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:44.509695 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:44.509700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:44.509703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:44.510045 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:44.510807 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:44.510816 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:44.511369 (+ 553us) client_negotiation.cc:769] Sending connection context
0923 17:37:44.512765 (+ 1396us) client_negotiation.cc:240] Negotiation successful
0923 17:37:44.513375 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17988,"thread_start_us":18555,"threads_started":1}
I20260923 17:37:44.517879 1571 heartbeater.cc:347] Connected to a master server at 127.20.240.126:45003
I20260923 17:37:44.518221 1571 heartbeater.cc:464] Registering TS with master...
I20260923 17:37:44.519116 1571 heartbeater.cc:511] Master 127.20.240.126:45003 requested a full tablet report, sending...
I20260923 17:37:44.521966 1358 ts_manager.cc:194] Registered new tserver with Master: e0e1f02b5e7344e1af63a9981b64c619 (127.20.240.65:46453)
I20260923 17:37:44.523972 1358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:57821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:37:44.539350 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.240.84:40793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:45003
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:37:44 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185063, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
W20260923 17:37:45.031332 1577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:37:45.031822 1577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:37:45.032007 1577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:37:45.043694 1577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:37:45.043924 1577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:37:45.044159 1577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:37:45.059012 1577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.240.84:40793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:45003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:37:45.061234 1577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:37:45.063139 1577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:37:45.102964 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:45.103899 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:37:45.109740 1577 server_base.cc:1034] running on GCE node
W20260923 17:37:45.110348 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:37:45.111238 1577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:37:45.112558 1577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:37:45.113842 1577 hybrid_clock.cc:648] HybridClock initialized: now 1790185065113768 us; error 59 us; skew 500 ppm
I20260923 17:37:45.114244 1577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:37:45 dist-test-slave-ppjl krb5kdc[881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185065, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:37:45.119426 1577 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:37:45.121738 1577 webserver.cc:460] Webserver started at http://127.20.240.66:43719/ using document root <none> and password file <none>
I20260923 17:37:45.122700 1577 fs_manager.cc:359] Metadata directory not provided
I20260923 17:37:45.122845 1577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:37:45.123250 1577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:37:45 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185065, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:37:45.147265 1577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "739a5826cbce4ddc8de1ecfd5160f839"
format_stamp: "Formatted at 2026-09-23 17:37:45 on dist-test-slave-ppjl"
server_key: "d907f248d734e42a02ee3666fa2c239a"
server_key_iv: "f75923de6101f8d953d8b24f596ff8b8"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:45.148523 1577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "739a5826cbce4ddc8de1ecfd5160f839"
format_stamp: "Formatted at 2026-09-23 17:37:45 on dist-test-slave-ppjl"
server_key: "d907f248d734e42a02ee3666fa2c239a"
server_key_iv: "f75923de6101f8d953d8b24f596ff8b8"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:45.168283 1577 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.013s sys 0.004s
I20260923 17:37:45.189693 1601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:45.191661 1577 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260923 17:37:45.191949 1577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "739a5826cbce4ddc8de1ecfd5160f839"
format_stamp: "Formatted at 2026-09-23 17:37:45 on dist-test-slave-ppjl"
server_key: "d907f248d734e42a02ee3666fa2c239a"
server_key_iv: "f75923de6101f8d953d8b24f596ff8b8"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:45.192327 1577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:37:45.215662 1577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:37:45.221410 1577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:37:45.221774 1577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:37:45.223001 1577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:37:45.224962 1577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:37:45.225060 1577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:45.225236 1577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:37:45.225304 1577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:45.278991 1577 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:43171
I20260923 17:37:45.279014 1714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:43171 every 8 connection(s)
I20260923 17:37:45.280831 1577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:37:45.284701 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 1577
I20260923 17:37:45.284880 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:37:45.285444 21441 external_mini_cluster.cc:1428] Setting key f32dd862fd1ece0028c41c4cd00609b0
Sep 23 17:37:45 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185065, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
I20260923 17:37:45.323793 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:45.287669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:59395 (local address 127.20.240.126:45003)
0923 17:37:45.295591 (+ 7922us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:45.295597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:45.301719 (+ 6122us) server_negotiation.cc:407] Connection header received
0923 17:37:45.304483 (+ 2764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:45.304490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:45.304608 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:45.304726 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:45.309717 (+ 4991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:45.310871 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:45.312924 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:45.313342 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:45.313465 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:45.317132 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:45.317139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:45.317143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:45.317189 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:45.319965 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:45.320774 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:45.320781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:45.320784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:45.320946 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:45.321457 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:45.321463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:45.321467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:45.321852 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:45.322036 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:45.323146 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0923 17:37:45.323416 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7774,"thread_start_us":9073,"threads_started":1}
I20260923 17:37:45.324942 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:45.284808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:45003 (local address 127.20.240.66:59395)
0923 17:37:45.301504 (+ 16696us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:45.301564 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:45.304099 (+ 2535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:45.307713 (+ 3614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:45.307729 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:45.308467 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:45.309435 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:45.309469 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:45.311132 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:45.311137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:45.312720 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:45.312736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:45.313599 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:45.313605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:45.313851 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:45.314657 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:45.314683 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:45.316820 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:45.320148 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:45.320159 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:45.320177 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:45.320630 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:45.321064 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:45.321069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:45.321074 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:45.321289 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:45.322037 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:45.322052 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:45.322536 (+ 484us) client_negotiation.cc:769] Sending connection context
0923 17:37:45.322911 (+ 375us) client_negotiation.cc:240] Negotiation successful
0923 17:37:45.323520 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16341,"thread_start_us":16120,"threads_started":1}
I20260923 17:37:45.326787 1715 heartbeater.cc:347] Connected to a master server at 127.20.240.126:45003
I20260923 17:37:45.327148 1715 heartbeater.cc:464] Registering TS with master...
I20260923 17:37:45.328047 1715 heartbeater.cc:511] Master 127.20.240.126:45003 requested a full tablet report, sending...
I20260923 17:37:45.331067 1358 ts_manager.cc:194] Registered new tserver with Master: 739a5826cbce4ddc8de1ecfd5160f839 (127.20.240.66:43171)
I20260923 17:37:45.332234 1358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:59395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:37:45.351037 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.240.84:40793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:45003
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:37:45.528231 1571 heartbeater.cc:503] Master 127.20.240.126:45003 was elected leader, sending a full tablet report...
W20260923 17:37:45.683820 1721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:37:45.684128 1721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:37:45.684244 1721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:37:45.691763 1721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:37:45.691937 1721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:37:45.692085 1721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:37:45.702311 1721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.240.84:40793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:45003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:37:45.703780 1721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:37:45.705122 1721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:37:45.718420 1727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:45.718456 1729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:45.718549 1726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:37:45.719848 1721 server_base.cc:1034] running on GCE node
I20260923 17:37:45.720489 1721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:37:45.721536 1721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:37:45.722780 1721 hybrid_clock.cc:648] HybridClock initialized: now 1790185065722707 us; error 54 us; skew 500 ppm
I20260923 17:37:45.723222 1721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:37:45 dist-test-slave-ppjl krb5kdc[881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185065, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:37:45.726421 1721 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:37:45.728075 1721 webserver.cc:460] Webserver started at http://127.20.240.67:43433/ using document root <none> and password file <none>
I20260923 17:37:45.728794 1721 fs_manager.cc:359] Metadata directory not provided
I20260923 17:37:45.728914 1721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:37:45.729207 1721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:37:45 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185065, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:37:45.745064 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "9d2a1f97cdd64a06858c384adf0a3969"
format_stamp: "Formatted at 2026-09-23 17:37:45 on dist-test-slave-ppjl"
server_key: "91bc6e204aa1cce3e1a339ee116229f2"
server_key_iv: "0e3962036251fd0165a937b640eb9cf7"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:45.745936 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "9d2a1f97cdd64a06858c384adf0a3969"
format_stamp: "Formatted at 2026-09-23 17:37:45 on dist-test-slave-ppjl"
server_key: "91bc6e204aa1cce3e1a339ee116229f2"
server_key_iv: "0e3962036251fd0165a937b640eb9cf7"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:45.752326 1721 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260923 17:37:45.768220 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:45.769809 1721 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:37:45.769994 1721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "9d2a1f97cdd64a06858c384adf0a3969"
format_stamp: "Formatted at 2026-09-23 17:37:45 on dist-test-slave-ppjl"
server_key: "91bc6e204aa1cce3e1a339ee116229f2"
server_key_iv: "0e3962036251fd0165a937b640eb9cf7"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:45.770211 1721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:37:45.791138 1721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:37:45.795328 1721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:37:45.795606 1721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:37:45.796567 1721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:37:45.797978 1721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:37:45.798054 1721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:45.798173 1721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:37:45.798228 1721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:45.830773 1721 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:39523
I20260923 17:37:45.830806 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:39523 every 8 connection(s)
I20260923 17:37:45.832239 1721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:37:45.836127 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 1721
I20260923 17:37:45.836293 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:37:45.836784 21441 external_mini_cluster.cc:1428] Setting key bb96440a608be6c9cb8913c43b4803d8
Sep 23 17:37:45 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185065, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:37:45.853279 1852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:45.834887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:39123 (local address 127.20.240.126:45003)
0923 17:37:45.835481 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:45.835486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:45.836528 (+ 1042us) server_negotiation.cc:407] Connection header received
0923 17:37:45.838660 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:45.838664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:45.838742 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:45.838882 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:45.840625 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:45.841324 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:45.843009 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:45.843491 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:45.843596 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:45.847083 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:45.847089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:45.847092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:45.847129 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:45.849698 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:45.850533 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:45.850538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:45.850541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:45.850712 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:45.851312 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:45.851317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:45.851320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:45.851700 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:45.851839 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:45.852761 (+ 922us) server_negotiation.cc:299] Negotiation successful
0923 17:37:45.852968 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":279,"threads_started":1}
I20260923 17:37:45.854336 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:45.835535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:45003 (local address 127.20.240.67:39123)
0923 17:37:45.836349 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:45.836404 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:45.838389 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:45.839096 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:45.839109 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:45.839741 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:45.840473 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:45.840493 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:45.841496 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:45.841500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:45.842823 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:45.842836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:45.843754 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:45.843760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:45.844024 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:45.844642 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:45.844664 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:45.846818 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:45.849845 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:45.849853 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:45.849868 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:45.850393 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:45.850834 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:45.850839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:45.850842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:45.851209 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:45.851827 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:45.851833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:45.852319 (+ 486us) client_negotiation.cc:769] Sending connection context
0923 17:37:45.852648 (+ 329us) client_negotiation.cc:240] Negotiation successful
0923 17:37:45.853182 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":334,"threads_started":1}
I20260923 17:37:45.855679 1850 heartbeater.cc:347] Connected to a master server at 127.20.240.126:45003
I20260923 17:37:45.855994 1850 heartbeater.cc:464] Registering TS with master...
I20260923 17:37:45.856680 1850 heartbeater.cc:511] Master 127.20.240.126:45003 requested a full tablet report, sending...
I20260923 17:37:45.858117 1358 ts_manager.cc:194] Registered new tserver with Master: 9d2a1f97cdd64a06858c384adf0a3969 (127.20.240.67:39523)
I20260923 17:37:45.858894 1358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:39123
I20260923 17:37:45.865665 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:37:45.867479 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 1438
Sep 23 17:37:45 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:37:45.885962 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:45.868611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54926 (local address 127.20.240.65:46453)
0923 17:37:45.869237 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:45.869245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:45.869271 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:37:45.869331 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:45.869338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:45.869558 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:45.869686 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:45.871113 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:45.872176 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:45.874109 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:45.874760 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:45.874885 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:45.879374 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:45.879396 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:45.879413 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:45.879462 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:45.882510 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:45.883486 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:45.883495 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:45.883498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:45.883667 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:45.884120 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:45.884130 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:45.884132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:45.884530 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:45.884773 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:45.885200 (+ 427us) server_negotiation.cc:299] Negotiation successful
0923 17:37:45.885476 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":310,"threads_started":1}
I20260923 17:37:45.887481 1566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:37:45.970691 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 1438
I20260923 17:37:45.983338 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 1577
Sep 23 17:37:45 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:37:46.000708 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:45.984259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.20.240.66:43171)
0923 17:37:45.984768 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:45.984775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:45.984800 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:37:45.984861 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:45.984867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:45.985075 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:45.985198 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:45.986505 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:45.987594 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:45.989014 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:45.989602 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:45.989706 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:45.994334 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:45.994372 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:45.994388 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:45.994432 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:45.997340 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:45.998321 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:45.998329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:45.998332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:45.998505 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:45.998946 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:45.998950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:45.998952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:45.999435 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:45.999690 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:46.000084 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 17:37:46.000323 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":259,"threads_started":1}
I20260923 17:37:46.002069 1710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:37:46.084322 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 1577
I20260923 17:37:46.097013 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 1721
Sep 23 17:37:46 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:37:46.114199 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:46.097994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.20.240.67:39523)
0923 17:37:46.098606 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:46.098613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:46.098641 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:37:46.098701 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:46.098707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:46.098921 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:46.099086 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:46.100431 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:46.101445 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:46.102763 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:46.103381 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:46.103487 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:46.107798 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:46.107812 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:46.107827 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:46.107865 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:46.110736 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:46.111978 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:46.111987 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:46.111989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:46.112150 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:46.112544 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:46.112548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:46.112550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:46.112997 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:46.113364 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:46.113559 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 17:37:46.113769 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":298,"threads_started":1}
I20260923 17:37:46.115554 1845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:37:46.202943 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 1721
I20260923 17:37:46.215950 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 1326
I20260923 17:37:46.217382 1388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:37:46.324517 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 1326
I20260923 17:37:46.346773 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:45003
--webserver_interface=127.20.240.126
--webserver_port=46853
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:45003
--ranger_config_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.240.84:40793/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.240.84:40793/kms with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:37:46.675912 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:37:46.676239 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:37:46.676332 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:37:46.683931 1862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:37:46.684060 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:37:46.684104 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:37:46.684158 1862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:37:46.684206 1862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:37:46.695186 1862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.240.84:40793/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:45003
--ranger_config_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:45003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=46853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:37:46.696523 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:37:46.697844 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:37:46.710529 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:46.710474 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:46.710556 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:37:46.711591 1862 server_base.cc:1034] running on GCE node
I20260923 17:37:46.712442 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:37:46.713996 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:37:46.715265 1862 hybrid_clock.cc:648] HybridClock initialized: now 1790185066715195 us; error 61 us; skew 500 ppm
I20260923 17:37:46.715655 1862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:37:46 dist-test-slave-ppjl krb5kdc[881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185066, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:37:46.719203 1862 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:37:46.720877 1862 webserver.cc:460] Webserver started at http://127.20.240.126:46853/ using document root <none> and password file <none>
I20260923 17:37:46.721575 1862 fs_manager.cc:359] Metadata directory not provided
I20260923 17:37:46.721704 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:37:46 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185066, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:37:46.786499 1862 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:37:46.790426 1879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:46.792145 1862 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:37:46.792342 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "3a455f6cb1df416db910571ea726ab17"
format_stamp: "Formatted at 2026-09-23 17:37:42 on dist-test-slave-ppjl"
server_key: "1a2efb0019c787c10e4666bc23a443c2"
server_key_iv: "a7a4bb1f82059beae69fca1a8a1ac34b"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:46.793357 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:37:46.813812 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:37:46.818241 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:37:46.818552 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:37:46.837363 1862 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:45003
I20260923 17:37:46.837384 1930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:45003 every 8 connection(s)
I20260923 17:37:46.839068 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:37:46.840610 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 1862
I20260923 17:37:46.841670 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.240.84:40793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:46453
--local_ip_for_outbound_sockets=127.20.240.65
--tserver_master_addrs=127.20.240.126:45003
--webserver_port=37475
--webserver_interface=127.20.240.65
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.240.84:40793/kms with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:37:46.849094 1931 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:37:46.853150 1931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17: Bootstrap starting.
I20260923 17:37:46.872655 1931 log.cc:824] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17: Log is configured to *not* fsync() on all Append() calls
I20260923 17:37:46.882088 1931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:37:46.882622 1931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17: Bootstrap complete.
I20260923 17:37:46.891705 1931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a455f6cb1df416db910571ea726ab17" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45003 } }
I20260923 17:37:46.892570 1931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a455f6cb1df416db910571ea726ab17, State: Initialized, Role: FOLLOWER
I20260923 17:37:46.893234 1931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a455f6cb1df416db910571ea726ab17" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45003 } }
I20260923 17:37:46.893440 1931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:37:46.893554 1931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:37:46.893723 1931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:37:46.897881 1931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a455f6cb1df416db910571ea726ab17" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45003 } }
I20260923 17:37:46.898459 1931 leader_election.cc:302] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a455f6cb1df416db910571ea726ab17; no voters:
I20260923 17:37:46.899044 1931 leader_election.cc:288] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:37:46.899214 1935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:37:46.901340 1931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:37:46.906173 1935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [term 2 LEADER]: Becoming Leader. State: Replica: 3a455f6cb1df416db910571ea726ab17, State: Running, Role: LEADER
I20260923 17:37:46.906832 1935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a455f6cb1df416db910571ea726ab17" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45003 } }
I20260923 17:37:46.909816 1936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3a455f6cb1df416db910571ea726ab17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a455f6cb1df416db910571ea726ab17" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45003 } } }
I20260923 17:37:46.910230 1936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [sys.catalog]: This master's current role is: LEADER
I20260923 17:37:46.910818 1937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a455f6cb1df416db910571ea726ab17. Latest consensus state: current_term: 2 leader_uuid: "3a455f6cb1df416db910571ea726ab17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a455f6cb1df416db910571ea726ab17" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45003 } } }
I20260923 17:37:46.911095 1937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17 [sys.catalog]: This master's current role is: LEADER
I20260923 17:37:46.913133 1949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:37:46.922801 1949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:37:46.923769 1931 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 17:37:46.924467 1949 catalog_manager.cc:1259] Loaded cluster ID: 66e46c9969284e02ab8904201b3a5044
I20260923 17:37:46.924646 1949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:37:46.938805 1949 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:37:46.941552 1949 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17: Loaded TSK: 0
I20260923 17:37:46.943575 1949 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:37:47.300307 1933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:37:47.300812 1933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:37:47.300971 1933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:37:47.311431 1933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:37:47.311645 1933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:37:47.311828 1933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:37:47.327040 1933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.240.84:40793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:46453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=37475
--tserver_master_addrs=127.20.240.126:45003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:37:47.328857 1933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:37:47.330358 1933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:37:47.346197 1969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:47.347064 1970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:47.355464 1972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:37:47.356178 1933 server_base.cc:1034] running on GCE node
I20260923 17:37:47.357102 1933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:37:47.358534 1933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:37:47.361972 1933 hybrid_clock.cc:648] HybridClock initialized: now 1790185067361297 us; error 651 us; skew 500 ppm
I20260923 17:37:47.362500 1933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:37:47 dist-test-slave-ppjl krb5kdc[881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185067, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:37:47.365962 1933 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:37:47.367925 1933 webserver.cc:460] Webserver started at http://127.20.240.65:37475/ using document root <none> and password file <none>
I20260923 17:37:47.368829 1933 fs_manager.cc:359] Metadata directory not provided
I20260923 17:37:47.368980 1933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:37:47 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185067, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:37:47.392046 1933 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:37:47.394883 1980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:47.396377 1933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:37:47.396556 1933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "e0e1f02b5e7344e1af63a9981b64c619"
format_stamp: "Formatted at 2026-09-23 17:37:44 on dist-test-slave-ppjl"
server_key: "168b77152897537ccb7a692a1d867e43"
server_key_iv: "4b9018218aa96b78efeb79e83a2f340b"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:47.397647 1933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:37:47.429286 1933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:37:47.434996 1933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:37:47.435432 1933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:37:47.436774 1933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:37:47.441462 1933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:37:47.441645 1933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:47.441826 1933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:37:47.441960 1933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:47.511240 1933 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:46453
I20260923 17:37:47.511304 2094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:46453 every 8 connection(s)
I20260923 17:37:47.512844 1933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:37:47.522022 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 1933
I20260923 17:37:47.522888 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.240.84:40793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:43171
--local_ip_for_outbound_sockets=127.20.240.66
--tserver_master_addrs=127.20.240.126:45003
--webserver_port=43719
--webserver_interface=127.20.240.66
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.240.84:40793/kms with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:37:47 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185067, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:37:47.543534 2098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:47.516322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:45003 (local address 127.20.240.65:48731)
0923 17:37:47.517639 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:47.517683 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:47.519925 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:47.521824 (+ 1899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:47.521835 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:47.522412 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:47.523260 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:47.523285 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:47.525759 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:47.525765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:47.527914 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:47.527932 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:47.529492 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:47.529497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:47.529729 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:47.530468 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:47.530499 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:47.533368 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:47.538499 (+ 5131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:47.538517 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:47.538534 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:47.539057 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:47.539659 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:47.539664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:47.539668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:47.539934 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:47.540913 (+ 979us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:47.540921 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:47.541300 (+ 379us) client_negotiation.cc:769] Sending connection context
0923 17:37:47.541603 (+ 303us) client_negotiation.cc:240] Negotiation successful
0923 17:37:47.542158 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"thread_start_us":828,"threads_started":1}
I20260923 17:37:47.543633 2097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:47.516229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:48731 (local address 127.20.240.126:45003)
0923 17:37:47.517059 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:47.517081 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:47.517847 (+ 766us) server_negotiation.cc:407] Connection header received
0923 17:37:47.520669 (+ 2822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:47.520689 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:47.521208 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:47.521627 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:47.523556 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:47.525551 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:47.528097 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:47.528963 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:47.529112 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:47.533729 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:47.533741 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:47.533764 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:47.533818 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:47.537696 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:47.539267 (+ 1571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:47.539278 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:47.539286 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:47.539510 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:47.540079 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:47.540084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:47.540087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:47.540761 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:47.540977 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:47.541918 (+ 941us) server_negotiation.cc:299] Negotiation successful
0923 17:37:47.542349 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":264,"threads_started":1}
I20260923 17:37:47.547926 2095 heartbeater.cc:347] Connected to a master server at 127.20.240.126:45003
I20260923 17:37:47.548254 2095 heartbeater.cc:464] Registering TS with master...
I20260923 17:37:47.549130 2095 heartbeater.cc:511] Master 127.20.240.126:45003 requested a full tablet report, sending...
W20260923 17:37:47.552480 2095 heartbeater.cc:644] Failed to heartbeat to 127.20.240.126:45003 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:37:47.552718 2095 heartbeater.cc:464] Registering TS with master...
I20260923 17:37:47.553033 2095 heartbeater.cc:511] Master 127.20.240.126:45003 requested a full tablet report, sending...
I20260923 17:37:47.554319 2095 heartbeater.cc:464] Registering TS with master...
I20260923 17:37:47.554618 2095 heartbeater.cc:511] Master 127.20.240.126:45003 requested a full tablet report, sending...
W20260923 17:37:47.555832 2095 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:37:47.922025 2099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:37:47.922434 2099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:37:47.922580 2099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:37:47.933633 2099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:37:47.933790 2099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:37:47.934043 2099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:37:47.945247 2099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.240.84:40793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:43171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=43719
--tserver_master_addrs=127.20.240.126:45003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:37:47.946637 2099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:37:47.948451 2099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:37:47.964382 2112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:47.969142 2114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:47.970292 2116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:37:47.971016 2099 server_base.cc:1034] running on GCE node
I20260923 17:37:47.971971 2099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:37:47.973407 2099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:37:47.974677 2099 hybrid_clock.cc:648] HybridClock initialized: now 1790185067974565 us; error 96 us; skew 500 ppm
I20260923 17:37:47.975116 2099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:37:47 dist-test-slave-ppjl krb5kdc[881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185067, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:37:47.979400 2099 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:37:47.985726 2099 webserver.cc:460] Webserver started at http://127.20.240.66:43719/ using document root <none> and password file <none>
I20260923 17:37:47.986523 2099 fs_manager.cc:359] Metadata directory not provided
I20260923 17:37:47.986654 2099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:37:47 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185067, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:37:48.011826 2099 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:37:48.015877 2124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:48.017580 2099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:37:48.017786 2099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "739a5826cbce4ddc8de1ecfd5160f839"
format_stamp: "Formatted at 2026-09-23 17:37:45 on dist-test-slave-ppjl"
server_key: "d907f248d734e42a02ee3666fa2c239a"
server_key_iv: "f75923de6101f8d953d8b24f596ff8b8"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:48.019035 2099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:37:48.075210 2099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:37:48.080730 2099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:37:48.081125 2099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:37:48.088282 2099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:37:48.090167 2099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:37:48.090313 2099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:48.090427 2099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:37:48.090510 2099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:37:48 dist-test-slave-ppjl krb5kdc[881](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790185068, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:37:48.166383 2099 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:43171
I20260923 17:37:48.166481 2239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:43171 every 8 connection(s)
I20260923 17:37:48.168484 2099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:37:48.170871 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 2099
I20260923 17:37:48.171671 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.240.84:40793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:39523
--local_ip_for_outbound_sockets=127.20.240.67
--tserver_master_addrs=127.20.240.126:45003
--webserver_port=43433
--webserver_interface=127.20.240.67
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.240.84:40793/kms with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:37:48 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185067, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:37:48.208532 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:48.175478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:42135 (local address 127.20.240.126:45003)
0923 17:37:48.185825 (+ 10347us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:48.185832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:48.185863 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:37:48.186779 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:48.186786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:48.186887 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:48.187010 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:48.190458 (+ 3448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:48.191600 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:48.194016 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:48.194584 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:48.194701 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:48.200176 (+ 5475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:48.200184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:48.200189 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:48.200254 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:48.203905 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:48.205010 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:48.205017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:48.205021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:48.205187 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:48.205778 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:48.205784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:48.205788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:48.206187 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:48.206339 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:48.207793 (+ 1454us) server_negotiation.cc:299] Negotiation successful
0923 17:37:48.208144 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10197,"thread_start_us":10043,"threads_started":1}
I20260923 17:37:48.210749 2243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:48.176043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:45003 (local address 127.20.240.66:42135)
0923 17:37:48.177891 (+ 1848us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:48.177972 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:48.186394 (+ 8422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:48.187878 (+ 1484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:48.187893 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:48.188546 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:48.189378 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:48.189405 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:48.191809 (+ 2404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:48.191815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:48.193807 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:48.193824 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:48.195030 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:48.195036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:48.195345 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:48.196153 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:48.196183 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:48.198969 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:48.204123 (+ 5154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:48.204133 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:48.204155 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:48.204817 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:48.205327 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:48.205333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:48.205337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:48.205624 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:48.206420 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:48.206430 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:48.206933 (+ 503us) client_negotiation.cc:769] Sending connection context
0923 17:37:48.207267 (+ 334us) client_negotiation.cc:240] Negotiation successful
0923 17:37:48.209184 (+ 1917us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1506,"thread_start_us":916,"threads_started":1}
I20260923 17:37:48.212630 2240 heartbeater.cc:347] Connected to a master server at 127.20.240.126:45003
I20260923 17:37:48.212986 2240 heartbeater.cc:464] Registering TS with master...
I20260923 17:37:48.213835 2240 heartbeater.cc:511] Master 127.20.240.126:45003 requested a full tablet report, sending...
I20260923 17:37:48.216599 1896 ts_manager.cc:194] Registered new tserver with Master: 739a5826cbce4ddc8de1ecfd5160f839 (127.20.240.66:43171)
I20260923 17:37:48.218710 1896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:42135
Sep 23 17:37:48 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185068, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:37:48.555990 2095 heartbeater.cc:464] Registering TS with master...
I20260923 17:37:48.556492 2095 heartbeater.cc:511] Master 127.20.240.126:45003 requested a full tablet report, sending...
I20260923 17:37:48.557898 1896 ts_manager.cc:194] Registered new tserver with Master: e0e1f02b5e7344e1af63a9981b64c619 (127.20.240.65:46453)
I20260923 17:37:48.559113 1896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:48731
W20260923 17:37:48.671415 2244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:37:48.672000 2244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:37:48.672230 2244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:37:48.684109 2244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:37:48.684362 2244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:37:48.684614 2244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:37:48.700691 2244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.240.84:40793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:39523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=43433
--tserver_master_addrs=127.20.240.126:45003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:37:48.702818 2244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:37:48.704850 2244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:37:48.747769 2251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:48.749195 2255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:48.750305 2252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:37:48.750820 2244 server_base.cc:1034] running on GCE node
I20260923 17:37:48.751789 2244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:37:48.758142 2244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:37:48.759547 2244 hybrid_clock.cc:648] HybridClock initialized: now 1790185068759487 us; error 50 us; skew 500 ppm
I20260923 17:37:48.760047 2244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:37:48 dist-test-slave-ppjl krb5kdc[881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185068, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:37:48.764309 2244 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:37:48.799642 2244 webserver.cc:460] Webserver started at http://127.20.240.67:43433/ using document root <none> and password file <none>
I20260923 17:37:48.800801 2244 fs_manager.cc:359] Metadata directory not provided
I20260923 17:37:48.801074 2244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:37:48 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185068, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:37:48.878443 2244 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.000s sys 0.004s
I20260923 17:37:48.895145 2263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:48.897545 2244 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.000s sys 0.010s
I20260923 17:37:48.897861 2244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "9d2a1f97cdd64a06858c384adf0a3969"
format_stamp: "Formatted at 2026-09-23 17:37:45 on dist-test-slave-ppjl"
server_key: "91bc6e204aa1cce3e1a339ee116229f2"
server_key_iv: "0e3962036251fd0165a937b640eb9cf7"
server_key_version: "kuduclusterkey@0"
I20260923 17:37:48.899453 2244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 23 17:37:48 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185068, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for HTTP/127.20.240.84@KRBTEST.COM
I20260923 17:37:48.937716 2244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:37:48.944525 2244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:37:48.944841 2244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:37:48.950333 2244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:37:48.952368 2244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:37:48.952534 2244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260923 17:37:48.952651 2244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:37:48.952715 2244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:49.150661 2244 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:39523
I20260923 17:37:49.150686 2377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:39523 every 8 connection(s)
I20260923 17:37:49.152752 2244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:37:49.160429 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 2244
Sep 23 17:37:49 dist-test-slave-ppjl krb5kdc[881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185068, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:37:49.195442 2384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.163060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.20.240.126:45003)
0923 17:37:49.178297 (+ 15237us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:49.178302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:49.178326 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:37:49.178390 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:49.178394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:49.178475 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:49.178601 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:49.180965 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.181965 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.184863 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.185430 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.185550 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.188928 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:49.188935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:49.188939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:49.188981 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:49.192290 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:49.193173 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:49.193177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:49.193180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:49.193308 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:49.193750 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:49.193754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:49.193757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:49.194112 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:49.194348 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:49.194724 (+ 376us) server_negotiation.cc:299] Negotiation successful
0923 17:37:49.194976 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":15101,"thread_start_us":15633,"threads_started":1}
I20260923 17:37:49.196408 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.178894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:60591 (local address 127.20.240.126:45003)
0923 17:37:49.179467 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:49.179471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:49.179499 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:37:49.179568 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:49.179572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:49.179638 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:49.179744 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:49.181870 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.183136 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.184857 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.185728 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.185855 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.189825 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:49.189830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:49.189833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:49.189877 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:49.192710 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:49.193482 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:49.193487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:49.193490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:49.193602 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:49.194211 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:49.194215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:49.194219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:49.194602 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:49.194719 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:49.195815 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0923 17:37:49.196043 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":453,"thread_start_us":302,"threads_started":1}
I20260923 17:37:49.197485 2385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.166484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:45003 (local address 127.20.240.67:60591)
0923 17:37:49.170849 (+ 4365us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:49.170908 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:49.173314 (+ 2406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:49.180149 (+ 6835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:49.180160 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:49.180667 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:49.181595 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:49.181627 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.183371 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.183376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:49.184699 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:49.184714 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.185984 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.185991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:49.186288 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.186921 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:49.186952 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:49.189563 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:49.192927 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:49.192935 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:49.192951 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:49.193398 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:49.193721 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:49.193726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:49.193730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:49.194092 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:49.194763 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:49.194773 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:49.195384 (+ 611us) client_negotiation.cc:769] Sending connection context
0923 17:37:49.195720 (+ 336us) client_negotiation.cc:240] Negotiation successful
0923 17:37:49.196322 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4035,"thread_start_us":3789,"threads_started":1}
I20260923 17:37:49.199023 2379 heartbeater.cc:347] Connected to a master server at 127.20.240.126:45003
I20260923 17:37:49.199368 2379 heartbeater.cc:464] Registering TS with master...
I20260923 17:37:49.200246 2379 heartbeater.cc:511] Master 127.20.240.126:45003 requested a full tablet report, sending...
I20260923 17:37:49.201994 1896 ts_manager.cc:194] Registered new tserver with Master: 9d2a1f97cdd64a06858c384adf0a3969 (127.20.240.67:39523)
I20260923 17:37:49.202833 1896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:60591
I20260923 17:37:49.210963 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:37:49.223685 2240 heartbeater.cc:503] Master 127.20.240.126:45003 was elected leader, sending a full tablet report...
I20260923 17:37:49.244426 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.213980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35578 (local address 127.20.240.126:45003)
0923 17:37:49.214147 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:49.214152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:49.214614 (+ 462us) server_negotiation.cc:407] Connection header received
0923 17:37:49.214768 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:49.214772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:49.214856 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:49.214946 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:49.216476 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.217265 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.218666 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.219104 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.219260 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.238724 (+ 19464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:49.238730 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:49.238734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:49.238783 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:49.241566 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:49.242434 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:49.242439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:49.242442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:49.242579 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:49.242955 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:49.242959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:49.242961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:49.243310 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:49.243495 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:49.243884 (+ 389us) server_negotiation.cc:299] Negotiation successful
0923 17:37:49.244077 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:37:49.249501 1896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:37:49.256875 1896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:37:49.282078 2404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.274170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.20.240.67:39523)
0923 17:37:49.274903 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:49.274911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:49.274940 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:37:49.275026 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:49.275032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:49.275279 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:49.276116 (+ 837us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:37:49.276397 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.277538 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.279902 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.280905 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.281025 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.281277 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:49.281546 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 17:37:49.281719 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":434,"threads_started":1}
I20260923 17:37:49.282173 2403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.273674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:39523 (local address 127.0.0.1:58250)
0923 17:37:49.274240 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:49.274256 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:49.274376 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:49.275494 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:49.275498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:49.275523 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:37:49.275924 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:49.275937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.277663 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.277668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:49.279682 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:49.279700 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.281116 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.281122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:49.281309 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.281360 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 17:37:49.281486 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 17:37:49.281700 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":111,"thread_start_us":635,"threads_started":1}
I20260923 17:37:49.282620 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.271299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:43171 (local address 127.0.0.1:33400)
0923 17:37:49.272773 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:49.272823 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:49.273065 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:49.274343 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:49.274351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:49.274389 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:37:49.274781 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:49.274795 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.277011 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.277016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:49.279793 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:49.279809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.281739 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.281746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:49.281946 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.281983 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 17:37:49.282048 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 17:37:49.282195 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"mutex_wait_us":153,"thread_start_us":1498,"threads_started":1}
I20260923 17:37:49.282778 2401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.272140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46453 (local address 127.0.0.1:54942)
0923 17:37:49.273938 (+ 1798us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:49.273955 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:49.274087 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:49.274845 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:49.274849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:49.274872 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:37:49.275206 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:49.275217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.276843 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.276847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:49.279793 (+ 2946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:49.279809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.282004 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.282014 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:49.282188 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.282215 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:37:49.282290 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 17:37:49.282433 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1608,"mutex_wait_us":147,"thread_start_us":1476,"threads_started":1}
I20260923 17:37:49.283244 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.272524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.20.240.65:46453)
0923 17:37:49.273348 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:49.273356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:49.274219 (+ 863us) server_negotiation.cc:407] Connection header received
0923 17:37:49.274330 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:49.274337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:49.274581 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:49.274768 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:37:49.275397 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.276707 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.279995 (+ 3288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.281494 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.281622 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.281849 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:49.282736 (+ 887us) server_negotiation.cc:299] Negotiation successful
0923 17:37:49.282903 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":798,"threads_started":1}
I20260923 17:37:49.283747 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.271673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.20.240.66:43171)
0923 17:37:49.273678 (+ 2005us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:49.273688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:49.273717 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:37:49.273819 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:49.273827 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:49.274108 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:49.275009 (+ 901us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:37:49.275291 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.276863 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.280031 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.281494 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.282993 (+ 1499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.283206 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:49.283284 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 17:37:49.283442 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1845,"thread_start_us":1112,"threads_started":1}
I20260923 17:37:49.286237 2311 tablet_service.cc:1467] Processing CreateTablet for tablet 787b204d1ad04f1db497d62c706f1de0 (DEFAULT_TABLE table=test-table [id=46b85de209b845b193dca1e0e1e58b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:37:49.286942 2174 tablet_service.cc:1467] Processing CreateTablet for tablet 787b204d1ad04f1db497d62c706f1de0 (DEFAULT_TABLE table=test-table [id=46b85de209b845b193dca1e0e1e58b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:37:49.286940 2029 tablet_service.cc:1467] Processing CreateTablet for tablet 787b204d1ad04f1db497d62c706f1de0 (DEFAULT_TABLE table=test-table [id=46b85de209b845b193dca1e0e1e58b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:37:49.287521 2311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 787b204d1ad04f1db497d62c706f1de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:37:49.288319 2174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 787b204d1ad04f1db497d62c706f1de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:37:49.288348 2029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 787b204d1ad04f1db497d62c706f1de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:37:49.300310 2405 tablet_bootstrap.cc:492] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969: Bootstrap starting.
I20260923 17:37:49.304356 2405 tablet_bootstrap.cc:654] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969: Neither blocks nor log segments found. Creating new log.
I20260923 17:37:49.305451 2405 log.cc:824] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969: Log is configured to *not* fsync() on all Append() calls
I20260923 17:37:49.307014 2406 tablet_bootstrap.cc:492] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619: Bootstrap starting.
I20260923 17:37:49.309082 2405 tablet_bootstrap.cc:492] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969: No bootstrap required, opened a new log
I20260923 17:37:49.309365 2405 ts_tablet_manager.cc:1397] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 17:37:49.309885 2407 tablet_bootstrap.cc:492] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839: Bootstrap starting.
I20260923 17:37:49.310770 2406 tablet_bootstrap.cc:654] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619: Neither blocks nor log segments found. Creating new log.
I20260923 17:37:49.312105 2406 log.cc:824] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619: Log is configured to *not* fsync() on all Append() calls
I20260923 17:37:49.313009 2407 tablet_bootstrap.cc:654] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839: Neither blocks nor log segments found. Creating new log.
I20260923 17:37:49.314978 2407 log.cc:824] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839: Log is configured to *not* fsync() on all Append() calls
I20260923 17:37:49.315447 2406 tablet_bootstrap.cc:492] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619: No bootstrap required, opened a new log
I20260923 17:37:49.315778 2406 ts_tablet_manager.cc:1397] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 17:37:49.317514 2407 tablet_bootstrap.cc:492] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839: No bootstrap required, opened a new log
I20260923 17:37:49.317754 2407 ts_tablet_manager.cc:1397] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260923 17:37:49.320308 2405 raft_consensus.cc:348] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a5826cbce4ddc8de1ecfd5160f839" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 43171 } } peers { permanent_uuid: "9d2a1f97cdd64a06858c384adf0a3969" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39523 } } peers { permanent_uuid: "e0e1f02b5e7344e1af63a9981b64c619" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46453 } }
I20260923 17:37:49.320761 2405 raft_consensus.cc:374] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:37:49.320855 2405 raft_consensus.cc:729] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d2a1f97cdd64a06858c384adf0a3969, State: Initialized, Role: FOLLOWER
I20260923 17:37:49.321555 2405 consensus_queue.cc:260] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a5826cbce4ddc8de1ecfd5160f839" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 43171 } } peers { permanent_uuid: "9d2a1f97cdd64a06858c384adf0a3969" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39523 } } peers { permanent_uuid: "e0e1f02b5e7344e1af63a9981b64c619" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46453 } }
I20260923 17:37:49.323064 2379 heartbeater.cc:503] Master 127.20.240.126:45003 was elected leader, sending a full tablet report...
I20260923 17:37:49.323807 2405 ts_tablet_manager.cc:1428] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260923 17:37:49.325918 2407 raft_consensus.cc:348] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a5826cbce4ddc8de1ecfd5160f839" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 43171 } } peers { permanent_uuid: "9d2a1f97cdd64a06858c384adf0a3969" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39523 } } peers { permanent_uuid: "e0e1f02b5e7344e1af63a9981b64c619" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46453 } }
I20260923 17:37:49.326393 2407 raft_consensus.cc:374] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:37:49.326499 2407 raft_consensus.cc:729] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739a5826cbce4ddc8de1ecfd5160f839, State: Initialized, Role: FOLLOWER
I20260923 17:37:49.327135 2407 consensus_queue.cc:260] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a5826cbce4ddc8de1ecfd5160f839" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 43171 } } peers { permanent_uuid: "9d2a1f97cdd64a06858c384adf0a3969" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39523 } } peers { permanent_uuid: "e0e1f02b5e7344e1af63a9981b64c619" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46453 } }
I20260923 17:37:49.327055 2406 raft_consensus.cc:348] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a5826cbce4ddc8de1ecfd5160f839" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 43171 } } peers { permanent_uuid: "9d2a1f97cdd64a06858c384adf0a3969" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39523 } } peers { permanent_uuid: "e0e1f02b5e7344e1af63a9981b64c619" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46453 } }
I20260923 17:37:49.327503 2406 raft_consensus.cc:374] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:37:49.327606 2406 raft_consensus.cc:729] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e1f02b5e7344e1af63a9981b64c619, State: Initialized, Role: FOLLOWER
I20260923 17:37:49.328166 2406 consensus_queue.cc:260] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a5826cbce4ddc8de1ecfd5160f839" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 43171 } } peers { permanent_uuid: "9d2a1f97cdd64a06858c384adf0a3969" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39523 } } peers { permanent_uuid: "e0e1f02b5e7344e1af63a9981b64c619" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46453 } }
I20260923 17:37:49.329187 2095 heartbeater.cc:503] Master 127.20.240.126:45003 was elected leader, sending a full tablet report...
I20260923 17:37:49.329914 2406 ts_tablet_manager.cc:1428] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20260923 17:37:49.331576 2407 ts_tablet_manager.cc:1428] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
W20260923 17:37:49.420218 2383 tablet.cc:2369] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:37:49.424616 2241 tablet.cc:2369] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:37:49.439719 2412 raft_consensus.cc:482] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:37:49.440022 2412 raft_consensus.cc:504] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a5826cbce4ddc8de1ecfd5160f839" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 43171 } } peers { permanent_uuid: "9d2a1f97cdd64a06858c384adf0a3969" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39523 } } peers { permanent_uuid: "e0e1f02b5e7344e1af63a9981b64c619" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46453 } }
I20260923 17:37:49.441872 2412 leader_election.cc:288] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d2a1f97cdd64a06858c384adf0a3969 (127.20.240.67:39523), e0e1f02b5e7344e1af63a9981b64c619 (127.20.240.65:46453)
I20260923 17:37:49.451040 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.442337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:49117 (local address 127.20.240.65:46453)
0923 17:37:49.442552 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:49.442558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:49.443401 (+ 843us) server_negotiation.cc:407] Connection header received
0923 17:37:49.443610 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:49.443615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:49.443702 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:49.443807 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:37:49.444535 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.445723 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.448639 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.449881 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.450058 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.450136 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:49.450646 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 17:37:49.450784 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:37:49.451086 2404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.442427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:60275 (local address 127.20.240.67:39523)
0923 17:37:49.442663 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:49.442668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:49.443396 (+ 728us) server_negotiation.cc:407] Connection header received
0923 17:37:49.443610 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:49.443615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:49.443705 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:49.443921 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:37:49.444535 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.445712 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.448638 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.449907 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.450058 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.450136 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:49.450646 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 17:37:49.450784 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260923 17:37:49.451325 2415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.442199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:39523 (local address 127.20.240.66:60275)
0923 17:37:49.443256 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:49.443276 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:49.443466 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:49.443983 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:49.443987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:49.444027 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:37:49.444404 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:49.444414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.445907 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.445913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:49.448453 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:49.448470 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.450198 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.450204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:49.450424 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.450483 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 17:37:49.450599 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 17:37:49.450871 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":215,"thread_start_us":725,"threads_started":1}
I20260923 17:37:49.451363 2414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.442262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46453 (local address 127.20.240.66:49117)
0923 17:37:49.443297 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:49.443312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:49.443481 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:49.443840 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:49.443845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:49.443870 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:37:49.444321 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:49.444338 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.445866 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.445871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:49.448461 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:49.448470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.450178 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.450186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:49.450442 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.450503 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 17:37:49.450604 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 17:37:49.450800 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":162,"spinlock_wait_cycles":49920,"thread_start_us":452,"threads_started":1}
I20260923 17:37:49.452651 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787b204d1ad04f1db497d62c706f1de0" candidate_uuid: "739a5826cbce4ddc8de1ecfd5160f839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2a1f97cdd64a06858c384adf0a3969" is_pre_election: true
I20260923 17:37:49.452651 2049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787b204d1ad04f1db497d62c706f1de0" candidate_uuid: "739a5826cbce4ddc8de1ecfd5160f839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e1f02b5e7344e1af63a9981b64c619" is_pre_election: true
I20260923 17:37:49.453231 2331 raft_consensus.cc:2454] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 739a5826cbce4ddc8de1ecfd5160f839 in term 0.
I20260923 17:37:49.453231 2049 raft_consensus.cc:2454] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 739a5826cbce4ddc8de1ecfd5160f839 in term 0.
I20260923 17:37:49.453971 2125 leader_election.cc:302] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 739a5826cbce4ddc8de1ecfd5160f839, e0e1f02b5e7344e1af63a9981b64c619; no voters:
I20260923 17:37:49.454350 2412 raft_consensus.cc:2781] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:37:49.454501 2412 raft_consensus.cc:482] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:37:49.454602 2412 raft_consensus.cc:3037] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:37:49.456858 2412 raft_consensus.cc:504] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a5826cbce4ddc8de1ecfd5160f839" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 43171 } } peers { permanent_uuid: "9d2a1f97cdd64a06858c384adf0a3969" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39523 } } peers { permanent_uuid: "e0e1f02b5e7344e1af63a9981b64c619" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46453 } }
I20260923 17:37:49.457454 2412 leader_election.cc:288] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [CANDIDATE]: Term 1 election: Requested vote from peers 9d2a1f97cdd64a06858c384adf0a3969 (127.20.240.67:39523), e0e1f02b5e7344e1af63a9981b64c619 (127.20.240.65:46453)
I20260923 17:37:49.458034 2049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787b204d1ad04f1db497d62c706f1de0" candidate_uuid: "739a5826cbce4ddc8de1ecfd5160f839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e1f02b5e7344e1af63a9981b64c619"
I20260923 17:37:49.458153 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787b204d1ad04f1db497d62c706f1de0" candidate_uuid: "739a5826cbce4ddc8de1ecfd5160f839" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d2a1f97cdd64a06858c384adf0a3969"
I20260923 17:37:49.458240 2049 raft_consensus.cc:3037] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:37:49.458379 2331 raft_consensus.cc:3037] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:37:49.460685 2331 raft_consensus.cc:2454] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 739a5826cbce4ddc8de1ecfd5160f839 in term 1.
I20260923 17:37:49.460801 2049 raft_consensus.cc:2454] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 739a5826cbce4ddc8de1ecfd5160f839 in term 1.
I20260923 17:37:49.461231 2126 leader_election.cc:302] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 739a5826cbce4ddc8de1ecfd5160f839, 9d2a1f97cdd64a06858c384adf0a3969; no voters:
I20260923 17:37:49.461555 2412 raft_consensus.cc:2781] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:37:49.462023 2412 raft_consensus.cc:686] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [term 1 LEADER]: Becoming Leader. State: Replica: 739a5826cbce4ddc8de1ecfd5160f839, State: Running, Role: LEADER
I20260923 17:37:49.462558 2412 consensus_queue.cc:237] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a5826cbce4ddc8de1ecfd5160f839" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 43171 } } peers { permanent_uuid: "9d2a1f97cdd64a06858c384adf0a3969" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39523 } } peers { permanent_uuid: "e0e1f02b5e7344e1af63a9981b64c619" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46453 } }
I20260923 17:37:49.467537 1895 catalog_manager.cc:5495] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 reported cstate change: term changed from 0 to 1, leader changed from <none> to 739a5826cbce4ddc8de1ecfd5160f839 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "739a5826cbce4ddc8de1ecfd5160f839" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "739a5826cbce4ddc8de1ecfd5160f839" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 43171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d2a1f97cdd64a06858c384adf0a3969" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0e1f02b5e7344e1af63a9981b64c619" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46453 } health_report { overall_health: UNKNOWN } } }
W20260923 17:37:49.516520 2096 tablet.cc:2369] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:37:49.523442 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.517973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.20.240.66:43171)
0923 17:37:49.518169 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:49.518176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:49.518225 (+ 49us) server_negotiation.cc:407] Connection header received
0923 17:37:49.518486 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:49.518490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:49.518580 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:49.518715 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:37:49.519090 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.519878 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.521227 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.521633 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.521739 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.522262 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:37:49.522837 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:37:49.522966 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:49.523067 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 17:37:49.523150 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:37:49.532342 2331 raft_consensus.cc:1261] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969 [term 1 FOLLOWER]: Refusing update from remote peer 739a5826cbce4ddc8de1ecfd5160f839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:37:49.532339 2049 raft_consensus.cc:1261] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619 [term 1 FOLLOWER]: Refusing update from remote peer 739a5826cbce4ddc8de1ecfd5160f839: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:37:49.533259 2412 consensus_queue.cc:1035] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0e1f02b5e7344e1af63a9981b64c619" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:37:49.533648 2416 consensus_queue.cc:1035] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d2a1f97cdd64a06858c384adf0a3969" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:37:49.544373 2419 mvcc.cc:205] Tried to move back new op lower bound from 7332598044793831424 to 7332598044523126784. Current Snapshot: MvccSnapshot[applied={T|T < 7332598044793831424}]
I20260923 17:37:49.560240 1896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:37:49.560523 1896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:37:49.564350 1896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a455f6cb1df416db910571ea726ab17: Sending DeleteTablet for 3 replicas of tablet 787b204d1ad04f1db497d62c706f1de0
I20260923 17:37:49.565750 2029 tablet_service.cc:1514] Processing DeleteTablet for tablet 787b204d1ad04f1db497d62c706f1de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:37:49 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:54942
I20260923 17:37:49.565726 2174 tablet_service.cc:1514] Processing DeleteTablet for tablet 787b204d1ad04f1db497d62c706f1de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:37:49 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:33400
I20260923 17:37:49.565793 2311 tablet_service.cc:1514] Processing DeleteTablet for tablet 787b204d1ad04f1db497d62c706f1de0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:37:49 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:58250
I20260923 17:37:49.575543 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 1933
I20260923 17:37:49.598481 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.576780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54954 (local address 127.20.240.65:46453)
0923 17:37:49.577003 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:49.577010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:49.578459 (+ 1449us) server_negotiation.cc:407] Connection header received
0923 17:37:49.578547 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:49.578553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:49.578646 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:49.578760 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:49.583285 (+ 4525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.584547 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.586634 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.587401 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.587549 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.590808 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:49.590830 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:49.590853 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:49.590895 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:49.594894 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:49.595782 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:49.595792 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:49.595795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:49.595974 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:49.596333 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:49.596338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:49.596342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:49.596850 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:49.597181 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:49.597471 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 17:37:49.597820 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:37:49.599689 2090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:37:49.602182 2431 tablet_replica.cc:330] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969: stopping tablet replica
I20260923 17:37:49.603528 2431 raft_consensus.cc:2229] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:37:49.605065 2431 raft_consensus.cc:2258] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:37:49.608961 2431 ts_tablet_manager.cc:1905] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:37:49.610647 2430 tablet_replica.cc:330] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839: stopping tablet replica
I20260923 17:37:49.611929 2430 raft_consensus.cc:2229] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:37:49.613456 2430 raft_consensus.cc:2258] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:37:49.617408 2430 ts_tablet_manager.cc:1905] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:37:49.617771 2431 ts_tablet_manager.cc:1918] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:37:49.617987 2431 log.cc:1196] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/787b204d1ad04f1db497d62c706f1de0
I20260923 17:37:49.618543 2431 ts_tablet_manager.cc:1939] T 787b204d1ad04f1db497d62c706f1de0 P 9d2a1f97cdd64a06858c384adf0a3969: Deleting consensus metadata
I20260923 17:37:49.620640 1881 catalog_manager.cc:4841] TS 9d2a1f97cdd64a06858c384adf0a3969 (127.20.240.67:39523): tablet 787b204d1ad04f1db497d62c706f1de0 (table test-table [id=46b85de209b845b193dca1e0e1e58b4f]) successfully deleted
I20260923 17:37:49.626282 2430 ts_tablet_manager.cc:1918] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:37:49.626564 2430 log.cc:1196] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/787b204d1ad04f1db497d62c706f1de0
I20260923 17:37:49.627151 2430 ts_tablet_manager.cc:1939] T 787b204d1ad04f1db497d62c706f1de0 P 739a5826cbce4ddc8de1ecfd5160f839: Deleting consensus metadata
I20260923 17:37:49.629283 1881 catalog_manager.cc:4841] TS 739a5826cbce4ddc8de1ecfd5160f839 (127.20.240.66:43171): tablet 787b204d1ad04f1db497d62c706f1de0 (table test-table [id=46b85de209b845b193dca1e0e1e58b4f]) successfully deleted
I20260923 17:37:49.707988 2429 tablet_replica.cc:330] T 787b204d1ad04f1db497d62c706f1de0 P e0e1f02b5e7344e1af63a9981b64c619: stopping tablet replica
I20260923 17:37:49.708190 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 1933
W20260923 17:37:49.722544 1880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.240.65:46453 (error 108)
I20260923 17:37:49.723364 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 2099
I20260923 17:37:49.724392 2401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.723843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46453 (local address 127.0.0.1:54958)
0923 17:37:49.724052 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:49.724183 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260923 17:37:49.724712 1880 catalog_manager.cc:4584] TS e0e1f02b5e7344e1af63a9981b64c619 (127.20.240.65:46453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 787b204d1ad04f1db497d62c706f1de0: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46453: connect: Connection refused (error 111)
I20260923 17:37:49.743219 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.724516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33410 (local address 127.20.240.66:43171)
0923 17:37:49.724681 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:49.724687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:49.724708 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:37:49.724771 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:49.724776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:49.724873 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:49.724988 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:49.726344 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.727671 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.729289 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.729984 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.730115 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.735875 (+ 5760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:49.735896 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:49.735918 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:49.735986 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:49.740118 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:49.740908 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:49.740920 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:49.740924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:49.741121 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:49.741502 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:49.741507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:49.741511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:49.741998 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:49.742307 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:49.742552 (+ 245us) server_negotiation.cc:299] Negotiation successful
0923 17:37:49.742806 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 17:37:49.744192 2235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:37:49.778172 2401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.777718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46453 (local address 127.0.0.1:54968)
0923 17:37:49.777913 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:49.778008 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 17:37:49.779295 2401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.778914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46453 (local address 127.0.0.1:54978)
0923 17:37:49.779054 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:49.779139 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 17:37:49.840183 2401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.839649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46453 (local address 127.0.0.1:54994)
0923 17:37:49.839912 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:49.840019 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 17:37:49.841372 2401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.840988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46453 (local address 127.0.0.1:55004)
0923 17:37:49.841141 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:49.841232 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 17:37:49.857880 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 2099
I20260923 17:37:49.873056 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 2244
I20260923 17:37:49.887317 2404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.873840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.20.240.67:39523)
0923 17:37:49.874067 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:49.874071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:49.874092 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:37:49.874301 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:49.874306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:49.874393 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:49.874480 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:49.875938 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.876674 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.877957 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:49.878346 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:49.878443 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:49.881164 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:49.881182 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:49.881199 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:49.881241 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:49.884409 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:49.885271 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:49.885278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:49.885281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:49.885459 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:49.885933 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:49.885937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:49.885939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:49.886311 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:49.886549 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:49.886816 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 17:37:49.886987 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260923 17:37:49.888096 2372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:37:49.922210 2401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.921702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46453 (local address 127.0.0.1:55020)
0923 17:37:49.921936 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:49.922037 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 17:37:49.923416 2401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:49.922952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46453 (local address 127.0.0.1:55032)
0923 17:37:49.923213 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:49.923292 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260923 17:37:49.976251 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 2244
I20260923 17:37:49.989281 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 1862
I20260923 17:37:49.990585 1926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:37:50.103952 2401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:50.103226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46453 (local address 127.0.0.1:55044)
0923 17:37:50.103600 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:50.103724 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214}
I20260923 17:37:50.103964 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 1862
I20260923 17:37:50.125525 21441 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 17:37:50.461229 21441 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 17:37:50.461429 21441 mini_ranger.cc:66] Stopping Ranger...
I20260923 17:37:50.797338 21441 mini_ranger.cc:68] Stopped Ranger
2026-09-23 17:37:50.797 UTC [911] LOG: received smart shutdown request
2026-09-23 17:37:50.801 UTC [911] LOG: background worker "logical replication launcher" (PID 920) exited with exit code 1
2026-09-23 17:37:50.801 UTC [915] LOG: shutting down
2026-09-23 17:37:51.001 UTC [911] LOG: database system is shut down
2026-09-23T17:37:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53621 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:37:51 dist-test-slave-ppjl krb5kdc[2445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:37:51 dist-test-slave-ppjl krb5kdc[2445](info): set up 2 sockets
Sep 23 17:37:51 dist-test-slave-ppjl krb5kdc[2445](info): commencing operation
krb5kdc: starting...
W20260923 17:37:52.136263 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:37:52 dist-test-slave-ppjl krb5kdc[2445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:37:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:37:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:37:52.363492 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:35903
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:35903,127.20.240.125:35747,127.20.240.124:38327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:37:52.693739 2458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:37:52.694053 2458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:37:52.694159 2458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:37:52.701647 2458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:37:52.701777 2458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:37:52.701839 2458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:37:52.701896 2458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:37:52.701939 2458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:37:52.711980 2458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:35903,127.20.240.125:35747,127.20.240.124:38327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:35903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:37:52.713294 2458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:37:52.714558 2458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:37:52.726194 2463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:52.726184 2464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:52.726303 2466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:37:52.727056 2458 server_base.cc:1034] running on GCE node
I20260923 17:37:52.727891 2458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:37:52.729423 2458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:37:52.730629 2458 hybrid_clock.cc:648] HybridClock initialized: now 1790185072730589 us; error 29 us; skew 500 ppm
I20260923 17:37:52.730970 2458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:37:52 dist-test-slave-ppjl krb5kdc[2445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185072, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:37:52.734261 2458 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:37:52.735886 2458 webserver.cc:460] Webserver started at http://127.20.240.126:40823/ using document root <none> and password file <none>
I20260923 17:37:52.736581 2458 fs_manager.cc:359] Metadata directory not provided
I20260923 17:37:52.736701 2458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:37:52.737003 2458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:37:52.740173 2458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "5394e16843bf4c059c352a66e333ddcf"
format_stamp: "Formatted at 2026-09-23 17:37:52 on dist-test-slave-ppjl"
server_key: "a089e50d20d9feaf85bd91a520214b2b"
server_key_iv: "f21648ae489b26720ef66ae754a6666b"
server_key_version: "encryptionkey@0"
I20260923 17:37:52.740964 2458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "5394e16843bf4c059c352a66e333ddcf"
format_stamp: "Formatted at 2026-09-23 17:37:52 on dist-test-slave-ppjl"
server_key: "a089e50d20d9feaf85bd91a520214b2b"
server_key_iv: "f21648ae489b26720ef66ae754a6666b"
server_key_version: "encryptionkey@0"
I20260923 17:37:52.746440 2458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260923 17:37:52.750049 2474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:52.751559 2458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:37:52.751740 2458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "5394e16843bf4c059c352a66e333ddcf"
format_stamp: "Formatted at 2026-09-23 17:37:52 on dist-test-slave-ppjl"
server_key: "a089e50d20d9feaf85bd91a520214b2b"
server_key_iv: "f21648ae489b26720ef66ae754a6666b"
server_key_version: "encryptionkey@0"
I20260923 17:37:52.752014 2458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:37:52.778270 2458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:37:52.782912 2458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:37:52.783227 2458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:37:52.801220 2458 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:35903
I20260923 17:37:52.801276 2525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:35903 every 8 connection(s)
I20260923 17:37:52.802611 2458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:37:52.805433 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 2458
I20260923 17:37:52.805646 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:37:52.806254 21441 external_mini_cluster.cc:1428] Setting key 8aa3cf270af3d485af97bb8f0a0b6101
I20260923 17:37:52.806526 2526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:37:52.813588 2526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.240.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.125@KRBTEST.COM" created.
Sep 23 17:37:52 dist-test-slave-ppjl krb5kdc[2445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185072, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:37:52.837999 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:52.817355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.20.240.126:35903)
0923 17:37:52.819223 (+ 1868us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:52.819231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:52.819295 (+ 64us) server_negotiation.cc:407] Connection header received
0923 17:37:52.820844 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:52.820850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:52.821123 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:52.821319 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:52.822676 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:52.823987 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:52.825378 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:52.825986 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:52.826185 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:52.830033 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:52.830047 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:52.830063 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:52.830103 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:52.833638 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:52.834381 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:52.834389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:52.834393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:52.834582 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:52.835006 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:52.835011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:52.835013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:52.835697 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:52.835890 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:52.836592 (+ 702us) server_negotiation.cc:299] Negotiation successful
0923 17:37:52.836863 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1607,"spinlock_wait_cycles":19328,"thread_start_us":1370,"threads_started":1}
I20260923 17:37:52.838049 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:52.817354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35903 (local address 127.0.0.1:35614)
0923 17:37:52.818301 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:52.818351 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:52.820445 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:52.821380 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:52.821387 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:52.822119 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:52.822480 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:52.822500 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:52.824131 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:52.824135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:52.825241 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:52.825254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:52.826230 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:52.826237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:52.826415 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:52.827099 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:52.827124 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:52.829690 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:52.833821 (+ 4131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:52.833832 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:52.833843 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:52.834247 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:52.834688 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:52.834691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:52.834694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:52.834849 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:52.835897 (+ 1048us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:52.835904 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:52.836174 (+ 270us) client_negotiation.cc:769] Sending connection context
0923 17:37:52.836466 (+ 292us) client_negotiation.cc:240] Negotiation successful
0923 17:37:52.837058 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":61,"thread_start_us":443,"threads_started":1}
I20260923 17:37:52.841113 2526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:37:52.842206 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:52.841747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:35747 (local address 127.0.0.1:56998)
0923 17:37:52.841981 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:52.842094 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260923 17:37:52.842461 2475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
I20260923 17:37:52.843964 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:52.843630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:35747 (local address 127.0.0.1:57000)
0923 17:37:52.843765 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:52.843835 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 17:37:52.844149 2526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.125:35747: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.125.keytab.
Entry for principal kudu/127.20.240.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.125.keytab.
I20260923 17:37:52.860708 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.125:35747
--webserver_interface=127.20.240.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.125.keytab
--principal=kudu/127.20.240.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:35903,127.20.240.125:35747,127.20.240.124:38327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:37:52.893483 2526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } attempt: 1
I20260923 17:37:52.894702 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:52.894237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:35747 (local address 127.0.0.1:57016)
0923 17:37:52.894449 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:52.894573 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 17:37:52.895664 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:52.895317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:35747 (local address 127.0.0.1:57032)
0923 17:37:52.895463 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:52.895533 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 17:37:52.895830 2526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.125:35747: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
I20260923 17:37:52.964087 2526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } attempt: 2
I20260923 17:37:52.965325 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:52.964743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:35747 (local address 127.0.0.1:57040)
0923 17:37:52.965067 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:52.965159 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260923 17:37:52.966424 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:52.966047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:35747 (local address 127.0.0.1:57048)
0923 17:37:52.966194 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:52.966275 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 17:37:52.966617 2526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.125:35747: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
I20260923 17:37:53.057928 2526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } attempt: 3
I20260923 17:37:53.059062 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.058582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:35747 (local address 127.0.0.1:57060)
0923 17:37:53.058799 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.058915 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 17:37:53.060231 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.059800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:35747 (local address 127.0.0.1:57070)
0923 17:37:53.060006 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.060078 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260923 17:37:53.060464 2526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.125:35747: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
W20260923 17:37:53.189527 2533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:37:53.189898 2533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:37:53.190032 2533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:37:53.197495 2533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:37:53.197635 2533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:37:53.197702 2533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:37:53.197780 2533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:37:53.197845 2533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:37:53.203719 2526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } attempt: 4
I20260923 17:37:53.204908 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.204372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:35747 (local address 127.0.0.1:57072)
0923 17:37:53.204671 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.204760 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260923 17:37:53.205869 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.205505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:35747 (local address 127.0.0.1:57088)
0923 17:37:53.205645 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.205738 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 17:37:53.206056 2526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.125:35747: Network error: Client connection negotiation failed: client connection to 127.20.240.125:35747: connect: Connection refused (error 111)
I20260923 17:37:53.208478 2533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:35903,127.20.240.125:35747,127.20.240.124:38327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.125.keytab
--principal=kudu/127.20.240.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.125:35747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.240.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:37:53.209899 2533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:37:53.211253 2533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:37:53.223240 2539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:53.223253 2538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:53.223136 2541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:37:53.224332 2533 server_base.cc:1034] running on GCE node
I20260923 17:37:53.225204 2533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:37:53.226649 2533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:37:53.227921 2533 hybrid_clock.cc:648] HybridClock initialized: now 1790185073227823 us; error 53 us; skew 500 ppm
I20260923 17:37:53.228291 2533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:37:53 dist-test-slave-ppjl krb5kdc[2445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185073, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:37:53.231604 2533 init.cc:377] Logged in from keytab as kudu/127.20.240.125@KRBTEST.COM (short username kudu)
I20260923 17:37:53.233244 2533 webserver.cc:460] Webserver started at http://127.20.240.125:41659/ using document root <none> and password file <none>
I20260923 17:37:53.234107 2533 fs_manager.cc:359] Metadata directory not provided
I20260923 17:37:53.234241 2533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:37:53.234573 2533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:37:53.237823 2533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/data/instance:
uuid: "71d01072677f4b89b715aae029ed5082"
format_stamp: "Formatted at 2026-09-23 17:37:53 on dist-test-slave-ppjl"
server_key: "3e584f220d4c36322d89eca9388dd5c8"
server_key_iv: "9836f9cff6479a461793bfa1c6033e57"
server_key_version: "encryptionkey@0"
I20260923 17:37:53.238636 2533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/wal/instance:
uuid: "71d01072677f4b89b715aae029ed5082"
format_stamp: "Formatted at 2026-09-23 17:37:53 on dist-test-slave-ppjl"
server_key: "3e584f220d4c36322d89eca9388dd5c8"
server_key_iv: "9836f9cff6479a461793bfa1c6033e57"
server_key_version: "encryptionkey@0"
I20260923 17:37:53.244231 2533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 17:37:53.247956 2548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:53.249380 2533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:37:53.249567 2533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/wal
uuid: "71d01072677f4b89b715aae029ed5082"
format_stamp: "Formatted at 2026-09-23 17:37:53 on dist-test-slave-ppjl"
server_key: "3e584f220d4c36322d89eca9388dd5c8"
server_key_iv: "9836f9cff6479a461793bfa1c6033e57"
server_key_version: "encryptionkey@0"
I20260923 17:37:53.249763 2533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:37:53.294118 2533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:37:53.298516 2533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:37:53.298796 2533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:37:53.316849 2533 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.125:35747
I20260923 17:37:53.316941 2599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.125:35747 every 8 connection(s)
I20260923 17:37:53.318259 2533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/data/info.pb
I20260923 17:37:53.322314 2600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:37:53.322580 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 2533
I20260923 17:37:53.322772 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-1/wal/instance
I20260923 17:37:53.323370 21441 external_mini_cluster.cc:1428] Setting key 1472650827661c1807a3c68312a7ffe2
I20260923 17:37:53.331655 2600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.240.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.124@KRBTEST.COM" created.
Sep 23 17:37:53 dist-test-slave-ppjl krb5kdc[2445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185073, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.125@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:37:53.354795 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.334570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.20.240.126:35903)
0923 17:37:53.334793 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:53.334799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:53.336106 (+ 1307us) server_negotiation.cc:407] Connection header received
0923 17:37:53.338125 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:53.338131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:53.338229 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:53.338347 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:53.340096 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.341160 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.342822 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.343528 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.343673 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.348405 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:53.348411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:53.348414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:53.348451 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:53.351252 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.352127 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.352132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.352135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.352276 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.352791 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.352796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.352798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.353164 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:53.353335 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:53.354318 (+ 983us) server_negotiation.cc:299] Negotiation successful
0923 17:37:53.354553 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:37:53.356077 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.335042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35903 (local address 127.0.0.1:35620)
0923 17:37:53.335916 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.335976 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:53.337822 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:53.338540 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:53.338549 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:53.339227 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:53.339867 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:53.339886 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.341363 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.341368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:53.342655 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:53.342669 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.343752 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.343757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:53.343963 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.344661 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:53.344685 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:53.348120 (+ 3435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:53.351436 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:53.351446 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:53.351468 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:53.351983 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:53.352450 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:53.352454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:53.352456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:53.352665 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:53.353300 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:53.353310 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:53.353786 (+ 476us) client_negotiation.cc:769] Sending connection context
0923 17:37:53.354163 (+ 377us) client_negotiation.cc:240] Negotiation successful
0923 17:37:53.354778 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":337,"threads_started":1}
I20260923 17:37:53.358071 2600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:37:53 dist-test-slave-ppjl krb5kdc[2445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185073, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.125@KRBTEST.COM for kudu/127.20.240.125@KRBTEST.COM
I20260923 17:37:53.372889 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.358696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:35747 (local address 127.0.0.1:57104)
0923 17:37:53.358928 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.358943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:53.359032 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:53.360120 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:53.360124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:53.360568 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:53.360704 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:53.360712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.362040 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.362045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:53.362889 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:53.362899 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.363749 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.363754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:53.363968 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.364522 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:53.364539 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:53.366518 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:53.370172 (+ 3654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:53.370178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:53.370183 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:53.370572 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:53.370996 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:53.370999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:53.371002 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:53.371149 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:53.371833 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:53.371838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:53.371977 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 17:37:53.372105 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 17:37:53.372416 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260923 17:37:53.372954 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.359010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57104 (local address 127.20.240.125:35747)
0923 17:37:53.359697 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:53.359704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:53.359728 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:37:53.359784 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:53.359789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:53.359974 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:53.360115 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:53.360855 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.361909 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.363016 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.363614 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.363725 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.366727 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:53.366740 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:53.366755 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:53.366794 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:53.370024 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.370712 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.370720 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.370722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.370850 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.371297 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.371300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.371302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.371685 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:53.371881 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:53.372279 (+ 398us) server_negotiation.cc:299] Negotiation successful
0923 17:37:53.372523 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":532,"thread_start_us":381,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.124.keytab.
Entry for principal kudu/127.20.240.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.124.keytab.
I20260923 17:37:53.375774 2600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:37:53.375850 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.124:38327
--webserver_interface=127.20.240.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.124.keytab
--principal=kudu/127.20.240.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:35903,127.20.240.125:35747,127.20.240.124:38327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:37:53.377096 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.376513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60266)
0923 17:37:53.376720 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.376880 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260923 17:37:53.377425 2551 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
I20260923 17:37:53.379127 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.378675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60268)
0923 17:37:53.378872 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.378973 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260923 17:37:53.379408 2600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:38327: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
I20260923 17:37:53.428768 2600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } attempt: 1
I20260923 17:37:53.429946 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.429490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60272)
0923 17:37:53.429676 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.429779 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 17:37:53.431053 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.430602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60278)
0923 17:37:53.430779 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.430888 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 17:37:53.431316 2600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:38327: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
I20260923 17:37:53.499577 2600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } attempt: 2
I20260923 17:37:53.500890 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.500302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60282)
0923 17:37:53.500539 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.500670 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260923 17:37:53.502022 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.501578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60286)
0923 17:37:53.501768 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.501873 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260923 17:37:53.502271 2600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:38327: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
I20260923 17:37:53.505300 2526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } attempt: 5
Sep 23 17:37:53 dist-test-slave-ppjl krb5kdc[2445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185072, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for kudu/127.20.240.125@KRBTEST.COM
I20260923 17:37:53.519634 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.505918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:35747 (local address 127.0.0.1:57110)
0923 17:37:53.506114 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.506134 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:53.506258 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:53.506614 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:53.506617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:53.507037 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:53.507252 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:53.507263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.508602 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.508607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:53.509469 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:53.509482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.510458 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.510463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:53.510609 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.511257 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:53.511275 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:53.513504 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:53.517363 (+ 3859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:53.517368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:53.517370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:53.517732 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:53.518133 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:53.518136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:53.518138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:53.518270 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:53.518854 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:53.518857 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:53.518995 (+ 138us) client_negotiation.cc:769] Sending connection context
0923 17:37:53.519115 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 17:37:53.519380 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 17:37:53.519801 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.506127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57110 (local address 127.20.240.125:35747)
0923 17:37:53.506324 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:53.506330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:53.506352 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:37:53.506413 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:53.506418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:53.506492 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:53.506595 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:53.507405 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.508457 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.509603 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.510281 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.510419 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.513700 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:53.513706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:53.513710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:53.513754 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:53.517178 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.517862 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.517869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.517872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.518017 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.518365 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.518369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.518372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.518731 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:53.518886 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:53.519268 (+ 382us) server_negotiation.cc:299] Negotiation successful
0923 17:37:53.519504 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:37:53.520857 2526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:37:53.521965 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.521520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60298)
0923 17:37:53.521703 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.521802 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 17:37:53.522816 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.522511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60304)
0923 17:37:53.522652 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.522716 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 17:37:53.523001 2526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:38327: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
I20260923 17:37:53.574255 2526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } attempt: 1
I20260923 17:37:53.575511 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.574977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60312)
0923 17:37:53.575216 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.575326 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 17:37:53.576421 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.576100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60326)
0923 17:37:53.576238 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.576308 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 17:37:53.576609 2526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:38327: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
I20260923 17:37:53.593494 2600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } attempt: 3
I20260923 17:37:53.594756 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.594182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60332)
0923 17:37:53.594439 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.594552 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 17:37:53.596097 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.595584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60334)
0923 17:37:53.595844 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.595953 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260923 17:37:53.596359 2600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:38327: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
I20260923 17:37:53.644807 2526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } attempt: 2
I20260923 17:37:53.646174 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.645534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60346)
0923 17:37:53.645830 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.645965 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 17:37:53.647614 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.647099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60350)
0923 17:37:53.647370 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.647472 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260923 17:37:53.647903 2526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:38327: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
W20260923 17:37:53.702956 2608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:37:53.703270 2608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:37:53.703401 2608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:37:53.710698 2608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:37:53.710819 2608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:37:53.710875 2608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:37:53.710958 2608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:37:53.711012 2608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:37:53.721208 2608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:35903,127.20.240.125:35747,127.20.240.124:38327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.124.keytab
--principal=kudu/127.20.240.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.124:38327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.240.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:37:53.722609 2608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:37:53.723963 2608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:37:53.735469 2616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:53.735487 2614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:53.735733 2613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:37:53.736529 2608 server_base.cc:1034] running on GCE node
I20260923 17:37:53.737378 2608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:37:53.738766 2608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:37:53.739571 2600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } attempt: 4
I20260923 17:37:53.739960 2608 hybrid_clock.cc:648] HybridClock initialized: now 1790185073739904 us; error 50 us; skew 500 ppm
I20260923 17:37:53.740305 2608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:37:53.740701 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.740169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60364)
0923 17:37:53.740428 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.740531 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 17:37:53.741740 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.741429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60372)
0923 17:37:53.741568 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.741639 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 17:37:53.741938 2600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:38327: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Sep 23 17:37:53 dist-test-slave-ppjl krb5kdc[2445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185073, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:37:53.743458 2608 init.cc:377] Logged in from keytab as kudu/127.20.240.124@KRBTEST.COM (short username kudu)
I20260923 17:37:53.744925 2608 webserver.cc:460] Webserver started at http://127.20.240.124:33417/ using document root <none> and password file <none>
I20260923 17:37:53.745608 2608 fs_manager.cc:359] Metadata directory not provided
I20260923 17:37:53.745721 2608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:37:53.746011 2608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:37:53.749015 2608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/data/instance:
uuid: "8e67dd9265ac421a966700a59325af12"
format_stamp: "Formatted at 2026-09-23 17:37:53 on dist-test-slave-ppjl"
server_key: "3a57432b330ab0499d23e274cc98d550"
server_key_iv: "5e7d6741915fc57493eb8350f420b263"
server_key_version: "encryptionkey@0"
I20260923 17:37:53.749790 2608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/wal/instance:
uuid: "8e67dd9265ac421a966700a59325af12"
format_stamp: "Formatted at 2026-09-23 17:37:53 on dist-test-slave-ppjl"
server_key: "3a57432b330ab0499d23e274cc98d550"
server_key_iv: "5e7d6741915fc57493eb8350f420b263"
server_key_version: "encryptionkey@0"
I20260923 17:37:53.754142 2526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } attempt: 3
I20260923 17:37:53.755206 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.754731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60378)
0923 17:37:53.754939 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.755024 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 17:37:53.755213 2608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:37:53.756228 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.755816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60386)
0923 17:37:53.756055 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.756129 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260923 17:37:53.756454 2526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:38327: Network error: Client connection negotiation failed: client connection to 127.20.240.124:38327: connect: Connection refused (error 111)
I20260923 17:37:53.758806 2623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:53.760260 2608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 17:37:53.760421 2608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/wal
uuid: "8e67dd9265ac421a966700a59325af12"
format_stamp: "Formatted at 2026-09-23 17:37:53 on dist-test-slave-ppjl"
server_key: "3a57432b330ab0499d23e274cc98d550"
server_key_iv: "5e7d6741915fc57493eb8350f420b263"
server_key_version: "encryptionkey@0"
I20260923 17:37:53.760597 2608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:37:53.786263 2608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:37:53.790678 2608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:37:53.790956 2608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:37:53.809316 2608 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.124:38327
I20260923 17:37:53.809334 2674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.124:38327 every 8 connection(s)
I20260923 17:37:53.810742 2608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/data/info.pb
I20260923 17:37:53.815106 2675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:37:53.817716 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 2608
I20260923 17:37:53.817960 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184865977475-21441-0/minicluster-data/master-2/wal/instance
I20260923 17:37:53.818492 21441 external_mini_cluster.cc:1428] Setting key 107d690119209a63b709c85ee6b2ff7a
I20260923 17:37:53.823837 2675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:37:53 dist-test-slave-ppjl krb5kdc[2445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.124@KRBTEST.COM
Sep 23 17:37:53 dist-test-slave-ppjl krb5kdc[2445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185073, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.124@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:37:53.842306 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.826953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.20.240.126:35903)
0923 17:37:53.827208 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:53.827214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:53.827859 (+ 645us) server_negotiation.cc:407] Connection header received
0923 17:37:53.828055 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:53.828060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:53.828140 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:53.828240 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:53.829370 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.830252 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.831865 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.832503 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.832641 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.836215 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:53.836221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:53.836225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:53.836264 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:53.839250 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.840012 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.840017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.840020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.840172 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.840558 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.840562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.840564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.840839 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:53.840970 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:53.841636 (+ 666us) server_negotiation.cc:299] Negotiation successful
0923 17:37:53.841895 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:37:53.843580 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.826840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35903 (local address 127.0.0.1:35622)
0923 17:37:53.827705 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.827748 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:53.827920 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:53.828293 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:53.828298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:53.828883 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:53.829196 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:53.829209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.830416 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.830421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:53.831713 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:53.831726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.832715 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.832719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:53.832839 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.833504 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:53.833528 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:53.835958 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:53.839394 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:53.839404 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:53.839415 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:53.839850 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:53.840281 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:53.840284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:53.840287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:53.840444 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:53.840959 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:53.840967 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:53.841217 (+ 250us) client_negotiation.cc:769] Sending connection context
0923 17:37:53.841499 (+ 282us) client_negotiation.cc:240] Negotiation successful
0923 17:37:53.842036 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":1,"thread_start_us":415,"threads_started":1}
I20260923 17:37:53.843598 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.820647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.20.240.124:38327)
0923 17:37:53.821592 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:53.821611 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:53.821666 (+ 55us) server_negotiation.cc:407] Connection header received
0923 17:37:53.824029 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:53.824061 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:53.824673 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:53.825162 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:53.826588 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.828292 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.829915 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.830794 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.830972 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.835909 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:53.835924 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:53.835951 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:53.835995 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:53.839080 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.839976 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.839985 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.839989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.840146 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.840553 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.840556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.840558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.841196 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:53.841501 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:53.841843 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 17:37:53.842151 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":612,"thread_start_us":371,"threads_started":1}
I20260923 17:37:53.845628 2675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:37:53 dist-test-slave-ppjl krb5kdc[2445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185073, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.124@KRBTEST.COM for kudu/127.20.240.125@KRBTEST.COM
Sep 23 17:37:53 dist-test-slave-ppjl krb5kdc[2445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:37:53.859730 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.846273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:35747 (local address 127.0.0.1:57124)
0923 17:37:53.846452 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.846487 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:53.846593 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:53.847058 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:53.847063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:53.847557 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:53.847715 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:53.847724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.848676 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.848680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:53.849599 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:53.849611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.850307 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.850313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:53.850517 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.851109 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:53.851127 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:53.853349 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:53.856941 (+ 3592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:53.856946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:53.856949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:53.857406 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:53.857903 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:53.857907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:53.857909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:53.858060 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:53.858639 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:53.858645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:53.858845 (+ 200us) client_negotiation.cc:769] Sending connection context
0923 17:37:53.859012 (+ 167us) client_negotiation.cc:240] Negotiation successful
0923 17:37:53.859341 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 17:37:53.859757 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.846451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.20.240.125:35747)
0923 17:37:53.846646 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:53.846652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:53.846671 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:37:53.846744 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:53.846748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:53.846826 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:53.847019 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:53.847872 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.848543 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.849733 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.850161 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.850271 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.853567 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:53.853572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:53.853576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:53.853628 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:53.856777 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.857534 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.857540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.857543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.857719 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.858185 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.858191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.858194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.858514 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:53.858629 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:53.859143 (+ 514us) server_negotiation.cc:299] Negotiation successful
0923 17:37:53.859432 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:37:53.860776 2675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:37:53.862677 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.847302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.20.240.126:35903)
0923 17:37:53.847504 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:53.847509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:53.847522 (+ 13us) server_negotiation.cc:407] Connection header received
0923 17:37:53.847564 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:53.847568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:53.847622 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:53.847701 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:53.848954 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.849733 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.851302 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.851736 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.851850 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.856457 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:53.856474 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:53.856479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:53.856530 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:53.859758 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.860645 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.860652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.860655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.860892 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.861309 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.861312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.861314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.861632 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:53.861939 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:53.862213 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 17:37:53.862395 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 17:37:53 dist-test-slave-ppjl krb5kdc[2445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185073, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.124@KRBTEST.COM for kudu/127.20.240.124@KRBTEST.COM
Sep 23 17:37:53 dist-test-slave-ppjl krb5kdc[2445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.125@KRBTEST.COM
I20260923 17:37:53.874578 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.861587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60406)
0923 17:37:53.861766 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.861786 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:53.861880 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:53.862967 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:53.862972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:53.863373 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:53.863509 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:53.863517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.864682 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.864687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:53.865638 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:53.865650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.866305 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.866310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:53.866478 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.867013 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:53.867030 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:53.868964 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:53.872159 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:53.872164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:53.872169 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:53.872582 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:53.872932 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:53.872937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:53.872940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:53.873066 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:53.873547 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:53.873551 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:53.873709 (+ 158us) client_negotiation.cc:769] Sending connection context
0923 17:37:53.873840 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 17:37:53.874164 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":1}
I20260923 17:37:53.874627 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.861779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.20.240.124:38327)
0923 17:37:53.862616 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:53.862620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:53.862634 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:37:53.862671 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:53.862674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:53.862726 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:53.862818 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:53.863632 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.864576 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.865758 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.866184 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.866283 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.869127 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:53.869132 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:53.869135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:53.869161 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:53.872029 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.872694 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.872699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.872703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.872831 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.873159 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.873164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.873167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.873439 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:53.873566 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:53.873932 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 17:37:53.874328 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":746}
I20260923 17:37:53.878464 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.864229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57126 (local address 127.20.240.125:35747)
0923 17:37:53.864386 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:53.864391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:53.864410 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:37:53.864560 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:53.864564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:53.864620 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:53.864702 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:53.865903 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.866592 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.867882 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.868290 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.868407 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.872816 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:53.872821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:53.872824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:53.872936 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:53.875586 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.876449 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.876454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.876456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.876614 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.877076 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.877081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.877083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.877380 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:53.877599 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:53.877965 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 17:37:53.878152 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:37:53.880824 2675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12: Bootstrap starting.
I20260923 17:37:53.883697 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:37:53.884375 2675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12: Neither blocks nor log segments found. Creating new log.
I20260923 17:37:53.885380 2675 log.cc:824] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12: Log is configured to *not* fsync() on all Append() calls
I20260923 17:37:53.888490 2675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12: No bootstrap required, opened a new log
I20260923 17:37:53.900251 2675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } }
I20260923 17:37:53.900720 2675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:37:53.900837 2675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e67dd9265ac421a966700a59325af12, State: Initialized, Role: FOLLOWER
I20260923 17:37:53.901647 2675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } }
I20260923 17:37:53.908303 2688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } } }
I20260923 17:37:53.908831 2688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:37:53.909941 2675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:37:53.910446 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.888718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.20.240.126:35903)
0923 17:37:53.888938 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:53.888945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:53.889664 (+ 719us) server_negotiation.cc:407] Connection header received
0923 17:37:53.889989 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:53.889996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:53.890086 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:53.890157 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:53.892888 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.894139 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.896429 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.897129 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.897237 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.902369 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:53.902378 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:53.902384 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:53.902447 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:53.906547 (+ 4100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.907769 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.907776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.907780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.908008 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.908769 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.908774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.908777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.909153 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:53.909336 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:53.909831 (+ 495us) server_negotiation.cc:299] Negotiation successful
0923 17:37:53.910086 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 17:37:53.916723 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.890046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57138 (local address 127.20.240.125:35747)
0923 17:37:53.890213 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:53.890219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:53.900019 (+ 9800us) server_negotiation.cc:407] Connection header received
0923 17:37:53.900179 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:53.900184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:53.900277 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:53.900383 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:53.902292 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.903386 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.905739 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.906389 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.906515 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.910588 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:53.910594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:53.910598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:53.910640 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:53.913858 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.914595 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.914601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.914605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.914754 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.915125 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.915130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.915133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.915471 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:53.915710 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:53.916007 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 17:37:53.916263 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:37:53.921566 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.888523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.20.240.124:38327)
0923 17:37:53.889105 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:53.889111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:53.895615 (+ 6504us) server_negotiation.cc:407] Connection header received
0923 17:37:53.895891 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:53.895896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:53.895984 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:53.896056 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:53.901785 (+ 5729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.902936 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.905950 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.906643 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.906744 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.913529 (+ 6785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:53.913536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:53.913541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:53.913592 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:53.917508 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.918923 (+ 1415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.918930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.918934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.919187 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.919694 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.919698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.919700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.920155 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:53.920441 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:53.920839 (+ 398us) server_negotiation.cc:299] Negotiation successful
0923 17:37:53.921112 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447}
W20260923 17:37:53.926856 2699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:37:53.927012 2699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:37:53.933730 2526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } attempt: 4
I20260923 17:37:53.924434 2608 master_runner.cc:186] Error getting master registration for 127.20.240.126:35903: OK, instance_id { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" instance_seqno: 1790185072796009 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:37:53.938527 2608 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 23 17:37:53 dist-test-slave-ppjl krb5kdc[2445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185072, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for kudu/127.20.240.124@KRBTEST.COM
I20260923 17:37:53.951556 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.934368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60424)
0923 17:37:53.934578 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:53.934599 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:53.934732 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:53.935311 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:53.935316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:53.935866 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:53.936096 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:53.936109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.937592 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.937598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:53.938939 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:53.938955 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.939986 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.939993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:53.940182 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.941295 (+ 1113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:53.941319 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:53.943498 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:53.947943 (+ 4445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:53.947948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:53.947951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:53.948429 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:53.949722 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:53.949727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:53.949731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:53.949932 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:53.950515 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:53.950519 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:53.950675 (+ 156us) client_negotiation.cc:769] Sending connection context
0923 17:37:53.950813 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 17:37:53.951125 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 17:37:53.951632 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:53.934398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.20.240.124:38327)
0923 17:37:53.934574 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:53.934578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:53.934656 (+ 78us) server_negotiation.cc:407] Connection header received
0923 17:37:53.934898 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:53.934904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:53.935012 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:53.935107 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:53.936270 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.937454 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.939098 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:53.939795 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:53.939935 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:53.943719 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:53.943727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:53.943732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:53.943780 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:53.947668 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.948649 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.948655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.948659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.949568 (+ 909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:53.950055 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:53.950059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:53.950061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:53.950382 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:53.950524 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:53.950946 (+ 422us) server_negotiation.cc:299] Negotiation successful
0923 17:37:53.951214 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:37:53.956977 2526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf: Bootstrap starting.
I20260923 17:37:53.960495 2526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf: Neither blocks nor log segments found. Creating new log.
I20260923 17:37:53.961609 2526 log.cc:824] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf: Log is configured to *not* fsync() on all Append() calls
I20260923 17:37:53.965090 2526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf: No bootstrap required, opened a new log
I20260923 17:37:53.973564 2526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } }
I20260923 17:37:53.973950 2526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:37:53.974068 2526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5394e16843bf4c059c352a66e333ddcf, State: Initialized, Role: FOLLOWER
I20260923 17:37:53.974676 2526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } }
I20260923 17:37:53.975744 2702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } } }
I20260923 17:37:53.976187 2702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:37:53.976833 2526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:37:53.986227 2458 master_runner.cc:186] Error getting master registration for 127.20.240.125:35747: OK, instance_id { permanent_uuid: "71d01072677f4b89b715aae029ed5082" instance_seqno: 1790185073311471 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:37:53.995756 2458 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 17:37:53.997543 2713 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:37:53.997689 2713 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260923 17:37:54.036648 2480 tablet.cc:2369] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:37:54.041164 2600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } attempt: 5
W20260923 17:37:54.043128 2629 tablet.cc:2369] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 17:37:54 dist-test-slave-ppjl krb5kdc[2445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185073, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.125@KRBTEST.COM for kudu/127.20.240.124@KRBTEST.COM
I20260923 17:37:54.057544 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:54.041845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:38327 (local address 127.0.0.1:60428)
0923 17:37:54.042188 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:54.042209 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:54.042347 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:54.042694 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:54.042699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:54.043266 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:54.043502 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:54.043517 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:54.044851 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:54.044856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:54.045730 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:54.045742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:54.046497 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:54.046503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:54.046691 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:54.047278 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:54.047296 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:54.049403 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:54.052985 (+ 3582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:54.052991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:54.052995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:54.053458 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:54.053882 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:54.053887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:54.053892 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:54.054101 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:54.054937 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:54.054942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:54.055101 (+ 159us) client_negotiation.cc:769] Sending connection context
0923 17:37:54.055277 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 17:37:54.055595 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260923 17:37:54.058283 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:54.042071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.20.240.124:38327)
0923 17:37:54.042235 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:54.042240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:54.042262 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:37:54.042481 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:54.042486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:54.042553 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:54.042636 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:54.043829 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:54.044688 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:54.045876 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:54.046342 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:54.046457 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:54.049613 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:54.049618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:54.049621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:54.049656 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:54.052799 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:54.053576 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:54.053582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:54.053585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:54.053766 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:54.054265 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:54.054270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:54.054273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:54.054730 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:54.054855 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:54.055884 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0923 17:37:54.058018 (+ 2134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:37:54.063223 2600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082: Bootstrap starting.
I20260923 17:37:54.066877 2600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082: Neither blocks nor log segments found. Creating new log.
I20260923 17:37:54.068094 2600 log.cc:824] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082: Log is configured to *not* fsync() on all Append() calls
I20260923 17:37:54.070957 2600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082: No bootstrap required, opened a new log
I20260923 17:37:54.081071 2600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } }
I20260923 17:37:54.081460 2600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:37:54.081574 2600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d01072677f4b89b715aae029ed5082, State: Initialized, Role: FOLLOWER
I20260923 17:37:54.082242 2600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } }
I20260923 17:37:54.083233 2716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } } }
I20260923 17:37:54.083706 2716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:37:54.084556 2600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:37:54.109733 2533 master_runner.cc:263] No leader master found from master 71d01072677f4b89b715aae029ed5082
I20260923 17:37:54.109920 2533 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 17:37:54.115093 2727 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:37:54.115379 2727 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:37:54.259733 2716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:37:54.260093 2716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } }
I20260923 17:37:54.261791 2716 leader_election.cc:288] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5394e16843bf4c059c352a66e333ddcf (127.20.240.126:35903), 8e67dd9265ac421a966700a59325af12 (127.20.240.124:38327)
I20260923 17:37:54.263092 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71d01072677f4b89b715aae029ed5082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e67dd9265ac421a966700a59325af12" is_pre_election: true
I20260923 17:37:54.263759 2650 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d01072677f4b89b715aae029ed5082 in term 0.
I20260923 17:37:54.265529 2501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71d01072677f4b89b715aae029ed5082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5394e16843bf4c059c352a66e333ddcf" is_pre_election: true
I20260923 17:37:54.266170 2501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d01072677f4b89b715aae029ed5082 in term 0.
I20260923 17:37:54.267603 2549 leader_election.cc:302] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5394e16843bf4c059c352a66e333ddcf, 71d01072677f4b89b715aae029ed5082; no voters:
I20260923 17:37:54.268540 2716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:37:54.268729 2716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:37:54.268838 2716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:37:54.277319 2716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } }
I20260923 17:37:54.281445 2501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71d01072677f4b89b715aae029ed5082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5394e16843bf4c059c352a66e333ddcf"
I20260923 17:37:54.281721 2501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:37:54.286310 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71d01072677f4b89b715aae029ed5082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e67dd9265ac421a966700a59325af12"
I20260923 17:37:54.286604 2650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:37:54.290012 2650 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d01072677f4b89b715aae029ed5082 in term 1.
I20260923 17:37:54.290637 2551 leader_election.cc:302] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d01072677f4b89b715aae029ed5082, 8e67dd9265ac421a966700a59325af12; no voters:
I20260923 17:37:54.292059 2501 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d01072677f4b89b715aae029ed5082 in term 1.
I20260923 17:37:54.294998 2716 leader_election.cc:288] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [CANDIDATE]: Term 1 election: Requested vote from peers 5394e16843bf4c059c352a66e333ddcf (127.20.240.126:35903), 8e67dd9265ac421a966700a59325af12 (127.20.240.124:38327)
I20260923 17:37:54.295573 2716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:37:54.296056 2716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [term 1 LEADER]: Becoming Leader. State: Replica: 71d01072677f4b89b715aae029ed5082, State: Running, Role: LEADER
I20260923 17:37:54.296923 2716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } }
W20260923 17:37:54.301535 2554 tablet.cc:2369] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:37:54.310786 2729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71d01072677f4b89b715aae029ed5082. Latest consensus state: current_term: 1 leader_uuid: "71d01072677f4b89b715aae029ed5082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } } }
I20260923 17:37:54.311496 2729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [sys.catalog]: This master's current role is: LEADER
I20260923 17:37:54.316535 2731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:37:54.319221 2731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:37:54.342408 2650 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12 [term 1 FOLLOWER]: Refusing update from remote peer 71d01072677f4b89b715aae029ed5082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:37:54.343900 2729 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:37:54.352065 2501 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf [term 1 FOLLOWER]: Refusing update from remote peer 71d01072677f4b89b715aae029ed5082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:37:54.353552 2729 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:37:54.380354 2702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71d01072677f4b89b715aae029ed5082. Latest consensus state: current_term: 1 leader_uuid: "71d01072677f4b89b715aae029ed5082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } } }
I20260923 17:37:54.380702 2702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:37:54.399694 2716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71d01072677f4b89b715aae029ed5082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } } }
I20260923 17:37:54.400096 2716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [sys.catalog]: This master's current role is: LEADER
I20260923 17:37:54.400473 2732 mvcc.cc:205] Tried to move back new op lower bound from 7332598064467746816 to 7332598064333074432. Current Snapshot: MvccSnapshot[applied={T|T < 7332598064467746816}]
I20260923 17:37:54.402329 2731 catalog_manager.cc:1347] Generated new cluster ID: 8d2c34bea66b4e16a39d2bea141c820c
I20260923 17:37:54.402443 2731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:37:54.419446 2702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71d01072677f4b89b715aae029ed5082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } } }
I20260923 17:37:54.419831 2702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:37:54.445333 2688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71d01072677f4b89b715aae029ed5082. Latest consensus state: current_term: 1 leader_uuid: "71d01072677f4b89b715aae029ed5082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } } }
I20260923 17:37:54.445645 2688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:37:54.449210 2729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71d01072677f4b89b715aae029ed5082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } } }
I20260923 17:37:54.449527 2729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082 [sys.catalog]: This master's current role is: LEADER
I20260923 17:37:54.458041 2733 mvcc.cc:205] Tried to move back new op lower bound from 7332598064467746816 to 7332598064333074432. Current Snapshot: MvccSnapshot[applied={T|T < 7332598064467746816}]
I20260923 17:37:54.458796 2731 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:37:54.460845 2731 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:37:54.475555 2688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71d01072677f4b89b715aae029ed5082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5394e16843bf4c059c352a66e333ddcf" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35903 } } peers { permanent_uuid: "71d01072677f4b89b715aae029ed5082" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 35747 } } peers { permanent_uuid: "8e67dd9265ac421a966700a59325af12" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 38327 } } }
I20260923 17:37:54.475847 2688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:37:54.487017 2731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71d01072677f4b89b715aae029ed5082: Generated new TSK 0
I20260923 17:37:54.488132 2731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:37:54.929116 2699 catalog_manager.cc:1259] Loaded cluster ID: 8d2c34bea66b4e16a39d2bea141c820c
I20260923 17:37:54.929225 2699 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12: loading cluster ID for follower catalog manager: success
I20260923 17:37:54.931900 2699 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12: acquiring CA information for follower catalog manager: success
I20260923 17:37:54.933938 2699 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e67dd9265ac421a966700a59325af12: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:37:54.999957 2713 catalog_manager.cc:1259] Loaded cluster ID: 8d2c34bea66b4e16a39d2bea141c820c
I20260923 17:37:55.000070 2713 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf: loading cluster ID for follower catalog manager: success
I20260923 17:37:55.002681 2713 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf: acquiring CA information for follower catalog manager: success
I20260923 17:37:55.004742 2713 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5394e16843bf4c059c352a66e333ddcf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:37:55.083600 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 2458
I20260923 17:37:55.084879 2521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:37:55.195013 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 2458
I20260923 17:37:55.212709 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 2533
I20260923 17:37:55.213793 2595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:37:55.322326 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 2533
I20260923 17:37:55.339103 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 2608
I20260923 17:37:55.340279 2670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:37:55.447434 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 2608
2026-09-23T17:37:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4334 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:37:55 dist-test-slave-ppjl krb5kdc[2749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:37:55 dist-test-slave-ppjl krb5kdc[2749](info): set up 2 sockets
Sep 23 17:37:55 dist-test-slave-ppjl krb5kdc[2749](info): commencing operation
krb5kdc: starting...
W20260923 17:37:56.468338 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:37:56 dist-test-slave-ppjl krb5kdc[2749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:37:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:37:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:37:56.696979 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:37737
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:43881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:37737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:37:57.032985 2762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:37:57.033303 2762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:37:57.033416 2762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:37:57.041497 2762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:37:57.041669 2762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:37:57.041751 2762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:37:57.041832 2762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:37:57.041889 2762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:37:57.052678 2762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:43881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:37737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:37737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:37:57.054026 2762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:37:57.055464 2762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:37:57.068228 2767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:57.068264 2770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:57.068190 2768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:37:57.069408 2762 server_base.cc:1034] running on GCE node
I20260923 17:37:57.070269 2762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:37:57.071879 2762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:37:57.073133 2762 hybrid_clock.cc:648] HybridClock initialized: now 1790185077073079 us; error 43 us; skew 500 ppm
I20260923 17:37:57.073561 2762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:37:57 dist-test-slave-ppjl krb5kdc[2749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185077, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:37:57.077085 2762 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:37:57.078758 2762 webserver.cc:460] Webserver started at http://127.20.240.126:45021/ using document root <none> and password file <none>
I20260923 17:37:57.079568 2762 fs_manager.cc:359] Metadata directory not provided
I20260923 17:37:57.079698 2762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:37:57.080070 2762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:37:57.083350 2762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "f48736901cfe4ee4accaec9c163003c1"
format_stamp: "Formatted at 2026-09-23 17:37:57 on dist-test-slave-ppjl"
server_key: "d9a1f00dbe5f08609c86990592bd4ccd"
server_key_iv: "9da40dd9647068780c8fa96e79908493"
server_key_version: "encryptionkey@0"
I20260923 17:37:57.084193 2762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "f48736901cfe4ee4accaec9c163003c1"
format_stamp: "Formatted at 2026-09-23 17:37:57 on dist-test-slave-ppjl"
server_key: "d9a1f00dbe5f08609c86990592bd4ccd"
server_key_iv: "9da40dd9647068780c8fa96e79908493"
server_key_version: "encryptionkey@0"
I20260923 17:37:57.090022 2762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 17:37:57.094360 2777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:57.096091 2762 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:37:57.096310 2762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "f48736901cfe4ee4accaec9c163003c1"
format_stamp: "Formatted at 2026-09-23 17:37:57 on dist-test-slave-ppjl"
server_key: "d9a1f00dbe5f08609c86990592bd4ccd"
server_key_iv: "9da40dd9647068780c8fa96e79908493"
server_key_version: "encryptionkey@0"
I20260923 17:37:57.096555 2762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:37:57.122380 2762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:37:57.127467 2762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:37:57.127794 2762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:37:57.147655 2762 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:37737
I20260923 17:37:57.147663 2828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:37737 every 8 connection(s)
I20260923 17:37:57.149411 2762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:37:57.149835 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 2762
I20260923 17:37:57.150051 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:37:57.150827 21441 external_mini_cluster.cc:1428] Setting key f38bda279475224ab6acb32fb89766e7
I20260923 17:37:57.154390 2829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:37:57 dist-test-slave-ppjl krb5kdc[2749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:37:57.166844 2829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1: Bootstrap starting.
I20260923 17:37:57.170692 2829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1: Neither blocks nor log segments found. Creating new log.
I20260923 17:37:57.171878 2829 log.cc:824] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1: Log is configured to *not* fsync() on all Append() calls
I20260923 17:37:57.174779 2832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:57.152863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.20.240.126:37737)
0923 17:37:57.153787 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:57.153801 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:57.153862 (+ 61us) server_negotiation.cc:407] Connection header received
0923 17:37:57.155735 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:57.155762 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:57.156189 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:57.156579 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:57.158203 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:57.159660 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:57.161038 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:57.161771 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:57.162026 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:57.166654 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:57.166665 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:57.166684 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:57.166732 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:57.170194 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:57.171119 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:57.171127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:57.171135 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:57.171375 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:57.171814 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:57.171818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:57.171820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:57.172445 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:57.172761 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:57.173253 (+ 492us) server_negotiation.cc:299] Negotiation successful
0923 17:37:57.173717 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":399,"threads_started":1}
I20260923 17:37:57.175064 2829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1: No bootstrap required, opened a new log
I20260923 17:37:57.179673 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:37:57.184290 2829 raft_consensus.cc:348] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48736901cfe4ee4accaec9c163003c1" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 37737 } }
I20260923 17:37:57.184753 2829 raft_consensus.cc:374] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:37:57.184864 2829 raft_consensus.cc:729] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48736901cfe4ee4accaec9c163003c1, State: Initialized, Role: FOLLOWER
I20260923 17:37:57.185451 2829 consensus_queue.cc:260] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48736901cfe4ee4accaec9c163003c1" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 37737 } }
I20260923 17:37:57.185680 2829 raft_consensus.cc:388] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:37:57.185807 2829 raft_consensus.cc:482] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:37:57.185987 2829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:37:57.188342 2829 raft_consensus.cc:504] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48736901cfe4ee4accaec9c163003c1" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 37737 } }
I20260923 17:37:57.188869 2829 leader_election.cc:302] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f48736901cfe4ee4accaec9c163003c1; no voters:
I20260923 17:37:57.189481 2829 leader_election.cc:288] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:37:57.189617 2834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:37:57.190318 2834 raft_consensus.cc:686] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [term 1 LEADER]: Becoming Leader. State: Replica: f48736901cfe4ee4accaec9c163003c1, State: Running, Role: LEADER
I20260923 17:37:57.190902 2834 consensus_queue.cc:237] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48736901cfe4ee4accaec9c163003c1" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 37737 } }
I20260923 17:37:57.191639 2829 sys_catalog.cc:564] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:37:57.193696 2835 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f48736901cfe4ee4accaec9c163003c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48736901cfe4ee4accaec9c163003c1" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 37737 } } }
I20260923 17:37:57.194203 2835 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:37:57.193789 2836 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f48736901cfe4ee4accaec9c163003c1. Latest consensus state: current_term: 1 leader_uuid: "f48736901cfe4ee4accaec9c163003c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48736901cfe4ee4accaec9c163003c1" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 37737 } } }
I20260923 17:37:57.195750 2836 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1 [sys.catalog]: This master's current role is: LEADER
I20260923 17:37:57.196287 2843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:37:57.201682 2843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:37:57.212424 2843 catalog_manager.cc:1347] Generated new cluster ID: 7a4a3d8c8ac5476ca7eca3c611759235
I20260923 17:37:57.212556 2843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:37:57.227069 2843 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:37:57.228571 2843 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:37:57.239787 2843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f48736901cfe4ee4accaec9c163003c1: Generated new TSK 0
I20260923 17:37:57.240875 2843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:37:57.293345 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 2762
I20260923 17:37:57.294659 2824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:37:57.392346 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 2762
2026-09-23T17:37:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1941 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:37:57 dist-test-slave-ppjl krb5kdc[2858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:37:57 dist-test-slave-ppjl krb5kdc[2858](info): set up 2 sockets
Sep 23 17:37:57 dist-test-slave-ppjl krb5kdc[2858](info): commencing operation
krb5kdc: starting...
W20260923 17:37:58.412118 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:37:58 dist-test-slave-ppjl krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:37:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:37:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:37:58.643591 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41471
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:41471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:37:58.987551 2871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:37:58.987862 2871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:37:58.987986 2871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:37:58.995592 2871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:37:58.995728 2871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:37:58.995800 2871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:37:58.995862 2871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:37:58.995920 2871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:37:59.006873 2871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:41471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:37:59.008287 2871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:37:59.009668 2871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:37:59.021842 2876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:59.021883 2879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:59.021844 2877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:37:59.022655 2871 server_base.cc:1034] running on GCE node
I20260923 17:37:59.023523 2871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:37:59.025002 2871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:37:59.026223 2871 hybrid_clock.cc:648] HybridClock initialized: now 1790185079026153 us; error 52 us; skew 500 ppm
I20260923 17:37:59.026587 2871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:37:59 dist-test-slave-ppjl krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185079, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:37:59.029707 2871 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:37:59.031270 2871 webserver.cc:460] Webserver started at http://127.20.240.126:41375/ using document root <none> and password file <none>
I20260923 17:37:59.031972 2871 fs_manager.cc:359] Metadata directory not provided
I20260923 17:37:59.032089 2871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:37:59.032436 2871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:37:59.035604 2871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "977113bfb73548ccb31b85199ab722fb"
format_stamp: "Formatted at 2026-09-23 17:37:59 on dist-test-slave-ppjl"
server_key: "adf5462740a427597bc2d0f85c100ea0"
server_key_iv: "e25fff446b76458267e12fc62778a571"
server_key_version: "encryptionkey@0"
I20260923 17:37:59.036404 2871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "977113bfb73548ccb31b85199ab722fb"
format_stamp: "Formatted at 2026-09-23 17:37:59 on dist-test-slave-ppjl"
server_key: "adf5462740a427597bc2d0f85c100ea0"
server_key_iv: "e25fff446b76458267e12fc62778a571"
server_key_version: "encryptionkey@0"
I20260923 17:37:59.042052 2871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:37:59.045864 2886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:59.047586 2871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:37:59.047771 2871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "977113bfb73548ccb31b85199ab722fb"
format_stamp: "Formatted at 2026-09-23 17:37:59 on dist-test-slave-ppjl"
server_key: "adf5462740a427597bc2d0f85c100ea0"
server_key_iv: "e25fff446b76458267e12fc62778a571"
server_key_version: "encryptionkey@0"
I20260923 17:37:59.047972 2871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:37:59.063755 2871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:37:59.068408 2871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:37:59.068691 2871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:37:59.089648 2871 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:41471
I20260923 17:37:59.089668 2938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:41471 every 8 connection(s)
I20260923 17:37:59.091284 2871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:37:59.095181 2939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:37:59.096486 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 2871
I20260923 17:37:59.096710 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:37:59.097554 21441 external_mini_cluster.cc:1428] Setting key 87df6c0d6a8e0d7351e8fad2763a248a
I20260923 17:37:59.108290 2939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: Bootstrap starting.
Sep 23 17:37:59 dist-test-slave-ppjl krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:37:59.111483 2939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: Neither blocks nor log segments found. Creating new log.
I20260923 17:37:59.112496 2939 log.cc:824] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: Log is configured to *not* fsync() on all Append() calls
I20260923 17:37:59.115267 2939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: No bootstrap required, opened a new log
I20260923 17:37:59.122363 2942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:59.099749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47776 (local address 127.20.240.126:41471)
0923 17:37:59.100500 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:59.100512 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:59.100557 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:37:59.102133 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:59.102155 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:59.102563 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:59.102906 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:59.104329 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:59.105659 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:59.106958 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:59.107673 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:59.107835 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:59.112003 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:59.112011 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:59.112026 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:59.112063 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:59.117130 (+ 5067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:59.118080 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:59.118090 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:59.118099 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:59.118336 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:59.118812 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:59.118817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:59.118821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:59.119627 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:59.119917 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:59.120460 (+ 543us) server_negotiation.cc:299] Negotiation successful
0923 17:37:59.120931 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":529,"thread_start_us":324,"threads_started":1}
I20260923 17:37:59.125181 2939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } }
I20260923 17:37:59.125630 2939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:37:59.125746 2939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 977113bfb73548ccb31b85199ab722fb, State: Initialized, Role: FOLLOWER
I20260923 17:37:59.126276 2939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } }
I20260923 17:37:59.126482 2939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:37:59.126608 2939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:37:59.126768 2939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:37:59.129227 2939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } }
I20260923 17:37:59.129712 2939 leader_election.cc:302] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 977113bfb73548ccb31b85199ab722fb; no voters:
I20260923 17:37:59.130298 2939 leader_election.cc:288] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:37:59.130410 2944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:37:59.131098 2944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 1 LEADER]: Becoming Leader. State: Replica: 977113bfb73548ccb31b85199ab722fb, State: Running, Role: LEADER
I20260923 17:37:59.131662 2944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } }
I20260923 17:37:59.132382 2939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:37:59.134444 2945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "977113bfb73548ccb31b85199ab722fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } } }
I20260923 17:37:59.134841 2945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [sys.catalog]: This master's current role is: LEADER
I20260923 17:37:59.135342 2946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 977113bfb73548ccb31b85199ab722fb. Latest consensus state: current_term: 1 leader_uuid: "977113bfb73548ccb31b85199ab722fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } } }
I20260923 17:37:59.135908 2946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [sys.catalog]: This master's current role is: LEADER
I20260923 17:37:59.136253 2953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:37:59.141609 2953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:37:59.150949 2953 catalog_manager.cc:1347] Generated new cluster ID: e2d91e9f5126447ab61865d19476d167
I20260923 17:37:59.151078 2953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:37:59.169253 2953 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:37:59.170578 2953 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:37:59.176256 2953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: Generated new TSK 0
I20260923 17:37:59.177194 2953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:37:59.236557 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41471
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:37:59.567988 2965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:37:59.568307 2965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:37:59.568449 2965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:37:59.575871 2965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:37:59.576009 2965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:37:59.576159 2965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:37:59.586103 2965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:37:59.587553 2965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:37:59.588913 2965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:37:59.602448 2970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:59.602418 2971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:37:59.602582 2973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:37:59.603401 2965 server_base.cc:1034] running on GCE node
I20260923 17:37:59.604094 2965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:37:59.605173 2965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:37:59.606402 2965 hybrid_clock.cc:648] HybridClock initialized: now 1790185079606337 us; error 62 us; skew 500 ppm
I20260923 17:37:59.606738 2965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:37:59 dist-test-slave-ppjl krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185079, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:37:59.613442 2965 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:37:59.615530 2965 webserver.cc:460] Webserver started at http://127.20.240.65:44015/ using document root <none> and password file <none>
I20260923 17:37:59.616273 2965 fs_manager.cc:359] Metadata directory not provided
I20260923 17:37:59.616413 2965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:37:59.616719 2965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:37:59.619974 2965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "eacbf1b49d4649a2b8fc10108bb20f7e"
format_stamp: "Formatted at 2026-09-23 17:37:59 on dist-test-slave-ppjl"
server_key: "ece5783e5050a7a1d148b70f6ce50eb3"
server_key_iv: "56cfdbdaa1754e34e008c9874418a002"
server_key_version: "encryptionkey@0"
I20260923 17:37:59.620800 2965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "eacbf1b49d4649a2b8fc10108bb20f7e"
format_stamp: "Formatted at 2026-09-23 17:37:59 on dist-test-slave-ppjl"
server_key: "ece5783e5050a7a1d148b70f6ce50eb3"
server_key_iv: "56cfdbdaa1754e34e008c9874418a002"
server_key_version: "encryptionkey@0"
I20260923 17:37:59.626479 2965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:37:59.630276 2980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:59.632042 2965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:37:59.632222 2965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "eacbf1b49d4649a2b8fc10108bb20f7e"
format_stamp: "Formatted at 2026-09-23 17:37:59 on dist-test-slave-ppjl"
server_key: "ece5783e5050a7a1d148b70f6ce50eb3"
server_key_iv: "56cfdbdaa1754e34e008c9874418a002"
server_key_version: "encryptionkey@0"
I20260923 17:37:59.632439 2965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:37:59.646086 2965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:37:59.650612 2965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:37:59.650909 2965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:37:59.652001 2965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:37:59.653604 2965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:37:59.653685 2965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:59.653803 2965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:37:59.653860 2965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:37:59.686180 2965 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:46557
I20260923 17:37:59.686218 3093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:46557 every 8 connection(s)
I20260923 17:37:59.687502 2965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:37:59.692657 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 2965
I20260923 17:37:59.692853 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:37:59.693468 21441 external_mini_cluster.cc:1428] Setting key c6cf52147a7a8d8bfb629d2546cf2499
Sep 23 17:37:59 dist-test-slave-ppjl krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185079, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:37:59.708196 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:59.690059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:36153 (local address 127.20.240.126:41471)
0923 17:37:59.690500 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0923 17:37:59.690505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:37:59.691461 (+ 956us) server_negotiation.cc:407] Connection header received
0923 17:37:59.693286 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:37:59.693292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:37:59.693385 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:37:59.693479 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:37:59.695221 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:59.696269 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:59.697914 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:37:59.698468 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:37:59.698593 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:59.702075 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:37:59.702081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:37:59.702084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:37:59.702119 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:37:59.704680 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:59.705571 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:59.705576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:59.705578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:59.705706 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:37:59.706217 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:37:59.706221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:37:59.706223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:37:59.706557 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:37:59.706767 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 17:37:59.707651 (+ 884us) server_negotiation.cc:299] Negotiation successful
0923 17:37:59.707922 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":229,"threads_started":1}
I20260923 17:37:59.709558 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:37:59.690548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:36153)
0923 17:37:59.691299 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0923 17:37:59.691349 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:37:59.693019 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:37:59.693737 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:37:59.693747 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:37:59.694408 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:37:59.695019 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:59.695040 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:59.696435 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:59.696440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:59.697759 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:37:59.697771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:37:59.698683 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:37:59.698689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:37:59.698941 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:37:59.699643 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:37:59.699666 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:37:59.701834 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:37:59.704841 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:59.704851 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:59.704869 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:59.705430 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:59.705827 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:37:59.705832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:37:59.705836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:37:59.706072 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:37:59.706692 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:37:59.706701 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:37:59.707192 (+ 491us) client_negotiation.cc:769] Sending connection context
0923 17:37:59.707515 (+ 323us) client_negotiation.cc:240] Negotiation successful
0923 17:37:59.708069 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":281,"threads_started":1}
I20260923 17:37:59.711272 3094 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41471
I20260923 17:37:59.711596 3094 heartbeater.cc:464] Registering TS with master...
I20260923 17:37:59.712389 3094 heartbeater.cc:511] Master 127.20.240.126:41471 requested a full tablet report, sending...
I20260923 17:37:59.714664 2903 ts_manager.cc:194] Registered new tserver with Master: eacbf1b49d4649a2b8fc10108bb20f7e (127.20.240.65:46557)
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
I20260923 17:37:59.716218 2903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:36153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:37:59.747773 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41471
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:38:00.072930 3100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:00.073222 3100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:00.073342 3100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:00.081002 3100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:00.081135 3100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:00.081278 3100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:38:00.091462 3100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:00.092821 3100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:00.094107 3100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:00.107642 3105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:00.107692 3106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:00.107717 3108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:00.108147 3100 server_base.cc:1034] running on GCE node
I20260923 17:38:00.109014 3100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:00.110100 3100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:00.111310 3100 hybrid_clock.cc:648] HybridClock initialized: now 1790185080111269 us; error 33 us; skew 500 ppm
I20260923 17:38:00.111639 3100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:00 dist-test-slave-ppjl krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185080, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:00.114974 3100 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:38:00.116595 3100 webserver.cc:460] Webserver started at http://127.20.240.66:36207/ using document root <none> and password file <none>
I20260923 17:38:00.117290 3100 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:00.117411 3100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:00.117695 3100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:38:00.120782 3100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "a255d4034a1247838478b91846b78652"
format_stamp: "Formatted at 2026-09-23 17:38:00 on dist-test-slave-ppjl"
server_key: "0e22905ad33450bdcdde0bf612ca2a2d"
server_key_iv: "e4d8ca6a13c3ea27ade1aeee27650b78"
server_key_version: "encryptionkey@0"
I20260923 17:38:00.121546 3100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "a255d4034a1247838478b91846b78652"
format_stamp: "Formatted at 2026-09-23 17:38:00 on dist-test-slave-ppjl"
server_key: "0e22905ad33450bdcdde0bf612ca2a2d"
server_key_iv: "e4d8ca6a13c3ea27ade1aeee27650b78"
server_key_version: "encryptionkey@0"
I20260923 17:38:00.126644 3100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:38:00.130404 3115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:00.132031 3100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:38:00.132215 3100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "a255d4034a1247838478b91846b78652"
format_stamp: "Formatted at 2026-09-23 17:38:00 on dist-test-slave-ppjl"
server_key: "0e22905ad33450bdcdde0bf612ca2a2d"
server_key_iv: "e4d8ca6a13c3ea27ade1aeee27650b78"
server_key_version: "encryptionkey@0"
I20260923 17:38:00.132434 3100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:00.151988 3100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:00.156306 3100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:00.156577 3100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:00.157630 3100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:38:00.159032 3100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:38:00.159112 3100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:00.159252 3100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:38:00.159308 3100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:00.189819 3100 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:44359
I20260923 17:38:00.189848 3228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:44359 every 8 connection(s)
I20260923 17:38:00.191114 3100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:38:00.201094 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 3100
I20260923 17:38:00.201277 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:38:00.201710 21441 external_mini_cluster.cc:1428] Setting key 2408ba70f91e7a97e7f421dc38e00007
Sep 23 17:38:00 dist-test-slave-ppjl krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185080, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:38:00.211359 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:00.193659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:51593 (local address 127.20.240.126:41471)
0923 17:38:00.193829 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:00.193834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:00.195195 (+ 1361us) server_negotiation.cc:407] Connection header received
0923 17:38:00.196987 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:00.196993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:00.197074 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:00.197161 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:00.198692 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.199472 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.201059 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.201564 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.201684 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:00.205514 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:00.205519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:00.205523 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:00.205556 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:00.208122 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:00.208933 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:00.208936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:00.208939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:00.209063 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:00.209496 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:00.209499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:00.209501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:00.209770 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:00.209906 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:00.210731 (+ 825us) server_negotiation.cc:299] Negotiation successful
0923 17:38:00.210969 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:38:00.212229 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:00.194168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:51593)
0923 17:38:00.194975 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:00.195017 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:00.196695 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:00.197405 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:00.197413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:00.197935 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:38:00.198511 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:00.198530 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.199627 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.199631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:00.200911 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:00.200924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.201798 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.201806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:00.202026 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:00.202667 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:38:00.202691 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:38:00.205262 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:38:00.208257 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:00.208266 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:00.208285 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:00.208837 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:00.209153 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:00.209157 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:00.209161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:00.209394 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:00.209886 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:38:00.209894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:38:00.210316 (+ 422us) client_negotiation.cc:769] Sending connection context
0923 17:38:00.210629 (+ 313us) client_negotiation.cc:240] Negotiation successful
0923 17:38:00.211082 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":298,"threads_started":1}
I20260923 17:38:00.213678 3229 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41471
I20260923 17:38:00.214013 3229 heartbeater.cc:464] Registering TS with master...
I20260923 17:38:00.214888 3229 heartbeater.cc:511] Master 127.20.240.126:41471 requested a full tablet report, sending...
I20260923 17:38:00.216614 2903 ts_manager.cc:194] Registered new tserver with Master: a255d4034a1247838478b91846b78652 (127.20.240.66:44359)
I20260923 17:38:00.217509 2903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:51593
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:38:00.254966 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41471
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:38:00.578383 3234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:00.578673 3234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:00.578804 3234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:00.586086 3234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:00.586193 3234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:00.586347 3234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:38:00.595956 3234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:00.597242 3234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:00.598579 3234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:00.611796 3239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:00.611871 3240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:00.611824 3242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:00.612512 3234 server_base.cc:1034] running on GCE node
I20260923 17:38:00.613085 3234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:00.614032 3234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:00.615265 3234 hybrid_clock.cc:648] HybridClock initialized: now 1790185080615209 us; error 52 us; skew 500 ppm
I20260923 17:38:00.615587 3234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:00 dist-test-slave-ppjl krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185080, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:00.618734 3234 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:38:00.620522 3234 webserver.cc:460] Webserver started at http://127.20.240.67:38179/ using document root <none> and password file <none>
I20260923 17:38:00.621181 3234 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:00.621281 3234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:00.621529 3234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:38:00.624452 3234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "93a1ff7e14d64d418e9b4751ac279532"
format_stamp: "Formatted at 2026-09-23 17:38:00 on dist-test-slave-ppjl"
server_key: "7ea654838722b2a081ed3059704990d4"
server_key_iv: "53a169c046b5eee3a7d3df0f56a229fb"
server_key_version: "encryptionkey@0"
I20260923 17:38:00.625196 3234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "93a1ff7e14d64d418e9b4751ac279532"
format_stamp: "Formatted at 2026-09-23 17:38:00 on dist-test-slave-ppjl"
server_key: "7ea654838722b2a081ed3059704990d4"
server_key_iv: "53a169c046b5eee3a7d3df0f56a229fb"
server_key_version: "encryptionkey@0"
I20260923 17:38:00.630261 3234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 17:38:00.633814 3249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:00.635360 3234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:38:00.635561 3234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "93a1ff7e14d64d418e9b4751ac279532"
format_stamp: "Formatted at 2026-09-23 17:38:00 on dist-test-slave-ppjl"
server_key: "7ea654838722b2a081ed3059704990d4"
server_key_iv: "53a169c046b5eee3a7d3df0f56a229fb"
server_key_version: "encryptionkey@0"
I20260923 17:38:00.635835 3234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:00.656508 3234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:00.660480 3234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:00.660720 3234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:00.661631 3234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:38:00.662905 3234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:38:00.662974 3234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:00.663065 3234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:38:00.663106 3234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:00.698086 3234 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:44211
I20260923 17:38:00.698133 3361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:44211 every 8 connection(s)
I20260923 17:38:00.699591 3234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:38:00.707142 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 3234
I20260923 17:38:00.707371 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:38:00.707809 21441 external_mini_cluster.cc:1428] Setting key 548c7ea9ad08988aabc71a735a63bafe
Sep 23 17:38:00 dist-test-slave-ppjl krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185080, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:38:00.719617 3094 heartbeater.cc:503] Master 127.20.240.126:41471 was elected leader, sending a full tablet report...
I20260923 17:38:00.720820 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:00.702366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:36695 (local address 127.20.240.126:41471)
0923 17:38:00.702562 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:00.702569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:00.703830 (+ 1261us) server_negotiation.cc:407] Connection header received
0923 17:38:00.705642 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:00.705649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:00.705774 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:00.705864 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:00.707673 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.708472 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.710584 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.711134 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.711288 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:00.714921 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:00.714927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:00.714930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:00.714966 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:00.717613 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:00.718364 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:00.718370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:00.718373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:00.718510 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:00.718940 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:00.718944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:00.718946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:00.719291 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:00.719442 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:00.720223 (+ 781us) server_negotiation.cc:299] Negotiation successful
0923 17:38:00.720469 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:38:00.721992 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:00.702793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:36695)
0923 17:38:00.703572 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:00.703618 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:00.705328 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:00.706142 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:00.706152 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:00.706705 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:38:00.707477 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:00.707503 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.708625 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.708630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:00.710414 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:00.710430 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.711380 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.711386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:00.711591 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:00.712247 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:38:00.712274 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:38:00.714623 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:38:00.717774 (+ 3151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:00.717784 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:00.717802 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:00.718236 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:00.718615 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:00.718620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:00.718624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:00.718832 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:00.719442 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:38:00.719451 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:38:00.719835 (+ 384us) client_negotiation.cc:769] Sending connection context
0923 17:38:00.720101 (+ 266us) client_negotiation.cc:240] Negotiation successful
0923 17:38:00.720666 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":292,"threads_started":1}
I20260923 17:38:00.723449 3362 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41471
I20260923 17:38:00.723798 3362 heartbeater.cc:464] Registering TS with master...
I20260923 17:38:00.724545 3362 heartbeater.cc:511] Master 127.20.240.126:41471 requested a full tablet report, sending...
I20260923 17:38:00.725963 2903 ts_manager.cc:194] Registered new tserver with Master: 93a1ff7e14d64d418e9b4751ac279532 (127.20.240.67:44211)
I20260923 17:38:00.726758 2903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:36695
I20260923 17:38:00.736536 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:38:00.753954 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:00.741332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.20.240.126:41471)
0923 17:38:00.741507 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:00.741512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:00.741876 (+ 364us) server_negotiation.cc:407] Connection header received
0923 17:38:00.742015 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:00.742019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:00.742089 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:00.742173 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:00.743552 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.744332 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.745662 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.746072 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.746168 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:00.748674 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:00.748679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:00.748683 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:00.748719 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:00.751267 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:00.751999 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:00.752005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:00.752009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:00.752156 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:00.752580 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:00.752585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:00.752588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:00.752965 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:00.753187 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:00.753516 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 17:38:00.753699 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:38:00.758159 2903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:38:00.765098 2903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:38:00.800920 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:00.777590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:44211 (local address 127.0.0.1:40542)
0923 17:38:00.788989 (+ 11399us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:00.789048 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:00.789266 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:00.793014 (+ 3748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:00.793024 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:00.793068 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:00.793600 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:00.793613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.795622 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.795627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:00.798210 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:00.798223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.799792 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.799798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:00.800028 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:00.800085 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 17:38:00.800221 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 17:38:00.800425 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11108,"mutex_wait_us":32,"thread_start_us":15579,"threads_started":1}
I20260923 17:38:00.801542 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:00.777736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:44359 (local address 127.0.0.1:41564)
0923 17:38:00.792971 (+ 15235us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:00.792991 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:00.793150 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:00.794151 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:00.794155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:00.794177 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:00.794630 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:00.794643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.796386 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.796391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:00.798780 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:00.798795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.800774 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.800781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:00.800948 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:00.800980 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:38:00.801053 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 17:38:00.801175 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15082,"spinlock_wait_cycles":7936,"thread_start_us":10805,"threads_started":1}
I20260923 17:38:00.801698 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:00.777900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.20.240.67:44211)
0923 17:38:00.792148 (+ 14248us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:00.792158 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:00.792198 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:38:00.792320 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:00.792341 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:00.792684 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:00.792936 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:00.793865 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.795476 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.798393 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.799594 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.799715 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:00.799966 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:00.801180 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0923 17:38:00.801364 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14079,"thread_start_us":13911,"threads_started":1}
I20260923 17:38:00.802583 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:00.778005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.20.240.66:44359)
0923 17:38:00.791437 (+ 13432us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:00.791449 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:00.793297 (+ 1848us) server_negotiation.cc:407] Connection header received
0923 17:38:00.793427 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:00.793435 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:00.793686 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:00.793904 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:00.794862 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.796212 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.798944 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.800414 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.800539 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:00.802089 (+ 1550us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:00.802151 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 17:38:00.802273 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13215,"thread_start_us":13445,"threads_started":1}
I20260923 17:38:00.802763 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:00.791897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46557 (local address 127.0.0.1:33004)
0923 17:38:00.794900 (+ 3003us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:00.794917 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:00.795033 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:00.795826 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:00.795829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:00.795852 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:00.796168 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:00.796175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.798056 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.798062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:00.799890 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:00.799905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.802038 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.802045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:00.802207 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:00.802238 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 17:38:00.802331 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 17:38:00.802463 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2869,"thread_start_us":2723,"threads_started":1}
I20260923 17:38:00.803891 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:00.789938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.20.240.65:46557)
0923 17:38:00.792526 (+ 2588us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:00.792533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:00.795034 (+ 2501us) server_negotiation.cc:407] Connection header received
0923 17:38:00.795188 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:00.795196 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:00.795421 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:00.795609 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:00.796377 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.797916 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.800106 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:00.801530 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:00.801661 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:00.801894 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:00.802410 (+ 516us) server_negotiation.cc:299] Negotiation successful
0923 17:38:00.802580 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2427,"thread_start_us":2266,"threads_started":1}
I20260923 17:38:00.806053 3297 tablet_service.cc:1467] Processing CreateTablet for tablet 7170c1e6cf2e40d887758aef2e8dde93 (DEFAULT_TABLE table=test-table [id=967265ce74f143f3a5a50e239fb62f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:38:00.806041 3163 tablet_service.cc:1467] Processing CreateTablet for tablet 7170c1e6cf2e40d887758aef2e8dde93 (DEFAULT_TABLE table=test-table [id=967265ce74f143f3a5a50e239fb62f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:38:00.807487 3163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7170c1e6cf2e40d887758aef2e8dde93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:38:00.807619 3297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7170c1e6cf2e40d887758aef2e8dde93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:38:00.807135 3028 tablet_service.cc:1467] Processing CreateTablet for tablet 7170c1e6cf2e40d887758aef2e8dde93 (DEFAULT_TABLE table=test-table [id=967265ce74f143f3a5a50e239fb62f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:38:00.808530 3028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7170c1e6cf2e40d887758aef2e8dde93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:38:00.818920 3380 tablet_bootstrap.cc:492] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652: Bootstrap starting.
I20260923 17:38:00.821591 3381 tablet_bootstrap.cc:492] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532: Bootstrap starting.
I20260923 17:38:00.822723 3380 tablet_bootstrap.cc:654] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652: Neither blocks nor log segments found. Creating new log.
I20260923 17:38:00.823107 3382 tablet_bootstrap.cc:492] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e: Bootstrap starting.
I20260923 17:38:00.824694 3380 log.cc:824] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:00.825058 3381 tablet_bootstrap.cc:654] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532: Neither blocks nor log segments found. Creating new log.
I20260923 17:38:00.826098 3381 log.cc:824] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:00.826922 3382 tablet_bootstrap.cc:654] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e: Neither blocks nor log segments found. Creating new log.
I20260923 17:38:00.827847 3380 tablet_bootstrap.cc:492] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652: No bootstrap required, opened a new log
I20260923 17:38:00.827992 3382 log.cc:824] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:00.828213 3380 ts_tablet_manager.cc:1397] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 17:38:00.829226 3381 tablet_bootstrap.cc:492] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532: No bootstrap required, opened a new log
I20260923 17:38:00.829531 3381 ts_tablet_manager.cc:1397] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 17:38:00.830500 3382 tablet_bootstrap.cc:492] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e: No bootstrap required, opened a new log
I20260923 17:38:00.830750 3382 ts_tablet_manager.cc:1397] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260923 17:38:00.838950 3382 raft_consensus.cc:348] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } } peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } }
I20260923 17:38:00.839085 3380 raft_consensus.cc:348] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } } peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } }
I20260923 17:38:00.839476 3382 raft_consensus.cc:374] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:38:00.839566 3382 raft_consensus.cc:729] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eacbf1b49d4649a2b8fc10108bb20f7e, State: Initialized, Role: FOLLOWER
I20260923 17:38:00.839586 3380 raft_consensus.cc:374] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:38:00.839691 3380 raft_consensus.cc:729] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a255d4034a1247838478b91846b78652, State: Initialized, Role: FOLLOWER
I20260923 17:38:00.839958 3381 raft_consensus.cc:348] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } } peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } }
I20260923 17:38:00.840399 3381 raft_consensus.cc:374] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:38:00.840250 3382 consensus_queue.cc:260] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } } peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } }
I20260923 17:38:00.840493 3381 raft_consensus.cc:729] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a1ff7e14d64d418e9b4751ac279532, State: Initialized, Role: FOLLOWER
I20260923 17:38:00.840394 3380 consensus_queue.cc:260] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } } peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } }
I20260923 17:38:00.841051 3381 consensus_queue.cc:260] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } } peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } }
I20260923 17:38:00.841661 3229 heartbeater.cc:503] Master 127.20.240.126:41471 was elected leader, sending a full tablet report...
I20260923 17:38:00.842314 3380 ts_tablet_manager.cc:1428] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260923 17:38:00.843935 3382 ts_tablet_manager.cc:1428] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260923 17:38:00.845525 3362 heartbeater.cc:503] Master 127.20.240.126:41471 was elected leader, sending a full tablet report...
I20260923 17:38:00.846143 3381 ts_tablet_manager.cc:1428] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532: Time spent starting tablet: real 0.016s user 0.008s sys 0.009s
W20260923 17:38:00.939788 3095 tablet.cc:2369] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:38:00.942950 3230 tablet.cc:2369] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:38:00.951418 3363 tablet.cc:2369] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:38:01.105875 3388 raft_consensus.cc:482] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:38:01.106156 3388 raft_consensus.cc:504] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } } peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } }
I20260923 17:38:01.107957 3388 leader_election.cc:288] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a255d4034a1247838478b91846b78652 (127.20.240.66:44359), eacbf1b49d4649a2b8fc10108bb20f7e (127.20.240.65:46557)
I20260923 17:38:01.114013 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:01.108375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:44359 (local address 127.20.240.67:34567)
0923 17:38:01.108576 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:01.108595 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:01.108739 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:01.109099 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:01.109104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:01.109134 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:01.109358 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:01.109371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.110590 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.110594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:01.112305 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:01.112321 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.113458 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.113462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:01.113591 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:01.113617 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:38:01.113688 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 17:38:01.113789 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 17:38:01.114123 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:01.108368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:34567 (local address 127.20.240.66:44359)
0923 17:38:01.108551 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:01.108555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:01.108743 (+ 188us) server_negotiation.cc:407] Connection header received
0923 17:38:01.108897 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:01.108902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:01.108967 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:01.109076 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:01.109539 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.110466 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.112494 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.113317 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.113482 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:01.113566 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:01.113736 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 17:38:01.113868 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:38:01.115374 3389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:01.108758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46557 (local address 127.20.240.67:37889)
0923 17:38:01.109654 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:01.109668 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:01.109767 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:01.110109 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:01.110112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:01.110133 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:01.110402 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:01.110411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.111497 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.111504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:01.113133 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:01.113145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.114712 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.114717 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:01.114855 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:01.114881 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:38:01.114957 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 17:38:01.115048 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":510,"threads_started":1}
I20260923 17:38:01.115458 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:01.109077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:37889 (local address 127.20.240.65:46557)
0923 17:38:01.109331 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:01.109337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:01.109762 (+ 425us) server_negotiation.cc:407] Connection header received
0923 17:38:01.109894 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:01.109899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:01.109980 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:01.110072 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:01.110573 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.111359 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.113308 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.114538 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.114659 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:01.114711 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:01.115040 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 17:38:01.115198 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:38:01.115413 3183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7170c1e6cf2e40d887758aef2e8dde93" candidate_uuid: "93a1ff7e14d64d418e9b4751ac279532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a255d4034a1247838478b91846b78652" is_pre_election: true
I20260923 17:38:01.116043 3183 raft_consensus.cc:2454] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93a1ff7e14d64d418e9b4751ac279532 in term 0.
I20260923 17:38:01.116503 3048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7170c1e6cf2e40d887758aef2e8dde93" candidate_uuid: "93a1ff7e14d64d418e9b4751ac279532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" is_pre_election: true
I20260923 17:38:01.116711 3253 leader_election.cc:302] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93a1ff7e14d64d418e9b4751ac279532, a255d4034a1247838478b91846b78652; no voters:
I20260923 17:38:01.116995 3048 raft_consensus.cc:2454] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93a1ff7e14d64d418e9b4751ac279532 in term 0.
I20260923 17:38:01.117129 3388 raft_consensus.cc:2781] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:38:01.117280 3388 raft_consensus.cc:482] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:38:01.117394 3388 raft_consensus.cc:3037] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:38:01.119737 3388 raft_consensus.cc:504] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } } peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } }
I20260923 17:38:01.120299 3388 leader_election.cc:288] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [CANDIDATE]: Term 1 election: Requested vote from peers a255d4034a1247838478b91846b78652 (127.20.240.66:44359), eacbf1b49d4649a2b8fc10108bb20f7e (127.20.240.65:46557)
I20260923 17:38:01.120729 3183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7170c1e6cf2e40d887758aef2e8dde93" candidate_uuid: "93a1ff7e14d64d418e9b4751ac279532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a255d4034a1247838478b91846b78652"
I20260923 17:38:01.120795 3048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7170c1e6cf2e40d887758aef2e8dde93" candidate_uuid: "93a1ff7e14d64d418e9b4751ac279532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e"
I20260923 17:38:01.120921 3183 raft_consensus.cc:3037] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:38:01.120985 3048 raft_consensus.cc:3037] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:38:01.123102 3048 raft_consensus.cc:2454] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a1ff7e14d64d418e9b4751ac279532 in term 1.
I20260923 17:38:01.123102 3183 raft_consensus.cc:2454] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93a1ff7e14d64d418e9b4751ac279532 in term 1.
I20260923 17:38:01.123530 3253 leader_election.cc:302] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93a1ff7e14d64d418e9b4751ac279532, eacbf1b49d4649a2b8fc10108bb20f7e; no voters:
I20260923 17:38:01.123901 3388 raft_consensus.cc:2781] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:38:01.124341 3388 raft_consensus.cc:686] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [term 1 LEADER]: Becoming Leader. State: Replica: 93a1ff7e14d64d418e9b4751ac279532, State: Running, Role: LEADER
I20260923 17:38:01.124883 3388 consensus_queue.cc:237] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } } peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } }
I20260923 17:38:01.129561 2902 catalog_manager.cc:5495] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 93a1ff7e14d64d418e9b4751ac279532 (127.20.240.67). New cstate: current_term: 1 leader_uuid: "93a1ff7e14d64d418e9b4751ac279532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } health_report { overall_health: UNKNOWN } } }
I20260923 17:38:01.172771 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:01.167322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.20.240.67:44211)
0923 17:38:01.167830 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:01.167835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:01.167851 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:38:01.167941 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:01.167945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:01.168020 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:01.168165 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:38:01.168553 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.169260 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.170641 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.171048 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.171191 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:01.171708 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:38:01.172217 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:38:01.172384 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:01.172430 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 17:38:01.172515 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401}
I20260923 17:38:01.180899 3048 raft_consensus.cc:1261] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e [term 1 FOLLOWER]: Refusing update from remote peer 93a1ff7e14d64d418e9b4751ac279532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:38:01.180932 3183 raft_consensus.cc:1261] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652 [term 1 FOLLOWER]: Refusing update from remote peer 93a1ff7e14d64d418e9b4751ac279532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:38:01.181845 3390 consensus_queue.cc:1035] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:38:01.182176 3388 consensus_queue.cc:1035] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:38:01.194056 3393 mvcc.cc:205] Tried to move back new op lower bound from 7332598092507271168 to 7332598092291391488. Current Snapshot: MvccSnapshot[applied={T|T < 7332598092507271168}]
I20260923 17:38:01.196473 3395 mvcc.cc:205] Tried to move back new op lower bound from 7332598092507271168 to 7332598092291391488. Current Snapshot: MvccSnapshot[applied={T|T < 7332598092507271168}]
I20260923 17:38:01.204591 3394 mvcc.cc:205] Tried to move back new op lower bound from 7332598092507271168 to 7332598092291391488. Current Snapshot: MvccSnapshot[applied={T|T < 7332598092507271168 or (T in {7332598092507271168})}]
I20260923 17:38:01.212908 2903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:38:01.213176 2903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:38:01.216622 2903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: Sending DeleteTablet for 3 replicas of tablet 7170c1e6cf2e40d887758aef2e8dde93
I20260923 17:38:01.217800 3163 tablet_service.cc:1514] Processing DeleteTablet for tablet 7170c1e6cf2e40d887758aef2e8dde93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:38:01 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:41564
I20260923 17:38:01.217805 3297 tablet_service.cc:1514] Processing DeleteTablet for tablet 7170c1e6cf2e40d887758aef2e8dde93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:38:01 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:40542
I20260923 17:38:01.219182 3028 tablet_service.cc:1514] Processing DeleteTablet for tablet 7170c1e6cf2e40d887758aef2e8dde93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:38:01 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:33004
I20260923 17:38:01.219791 3404 tablet_replica.cc:330] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532: stopping tablet replica
I20260923 17:38:01.220093 3405 tablet_replica.cc:330] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e: stopping tablet replica
I20260923 17:38:01.220221 3404 raft_consensus.cc:2229] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:38:01.220494 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 2965
I20260923 17:38:01.220464 3405 raft_consensus.cc:2229] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:38:01.220978 3405 raft_consensus.cc:2258] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:38:01.221554 3403 tablet_replica.cc:330] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652: stopping tablet replica
I20260923 17:38:01.221932 3403 raft_consensus.cc:2229] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:38:01.222410 3403 raft_consensus.cc:2258] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:38:01.222813 3404 raft_consensus.cc:2258] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:38:01.224283 3405 ts_tablet_manager.cc:1905] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:38:01.224673 3404 ts_tablet_manager.cc:1905] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:38:01.224673 3403 ts_tablet_manager.cc:1905] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:38:01.231277 3403 ts_tablet_manager.cc:1918] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:38:01.231279 3404 ts_tablet_manager.cc:1918] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:38:01.231462 3404 log.cc:1196] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/7170c1e6cf2e40d887758aef2e8dde93
I20260923 17:38:01.231462 3403 log.cc:1196] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/7170c1e6cf2e40d887758aef2e8dde93
I20260923 17:38:01.231616 3405 ts_tablet_manager.cc:1918] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:38:01.231797 3405 log.cc:1196] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/7170c1e6cf2e40d887758aef2e8dde93
I20260923 17:38:01.231981 3403 ts_tablet_manager.cc:1939] T 7170c1e6cf2e40d887758aef2e8dde93 P a255d4034a1247838478b91846b78652: Deleting consensus metadata
I20260923 17:38:01.232028 3404 ts_tablet_manager.cc:1939] T 7170c1e6cf2e40d887758aef2e8dde93 P 93a1ff7e14d64d418e9b4751ac279532: Deleting consensus metadata
I20260923 17:38:01.232326 3405 ts_tablet_manager.cc:1939] T 7170c1e6cf2e40d887758aef2e8dde93 P eacbf1b49d4649a2b8fc10108bb20f7e: Deleting consensus metadata
I20260923 17:38:01.233831 2890 catalog_manager.cc:4841] TS a255d4034a1247838478b91846b78652 (127.20.240.66:44359): tablet 7170c1e6cf2e40d887758aef2e8dde93 (table test-table [id=967265ce74f143f3a5a50e239fb62f4e]) successfully deleted
I20260923 17:38:01.233883 2888 catalog_manager.cc:4841] TS 93a1ff7e14d64d418e9b4751ac279532 (127.20.240.67:44211): tablet 7170c1e6cf2e40d887758aef2e8dde93 (table test-table [id=967265ce74f143f3a5a50e239fb62f4e]) successfully deleted
I20260923 17:38:01.234844 2890 catalog_manager.cc:4841] TS eacbf1b49d4649a2b8fc10108bb20f7e (127.20.240.65:46557): tablet 7170c1e6cf2e40d887758aef2e8dde93 (table test-table [id=967265ce74f143f3a5a50e239fb62f4e]) successfully deleted
Sep 23 17:38:01 dist-test-slave-ppjl krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:38:01.242368 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:01.221645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.20.240.65:46557)
0923 17:38:01.221802 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:01.221807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:01.221923 (+ 116us) server_negotiation.cc:407] Connection header received
0923 17:38:01.222192 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:01.222198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:01.222275 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:01.222361 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:01.224780 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.225867 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.227981 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.228596 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.228697 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:01.236321 (+ 7624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:01.236337 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:01.236356 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:01.236401 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:01.239437 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:01.240191 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:01.240200 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:01.240203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:01.240374 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:01.240784 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:01.240789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:01.240792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:01.241181 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:01.241506 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:01.241776 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 17:38:01.241999 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:38:01.243189 3089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:01.360318 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 2965
I20260923 17:38:01.381845 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 3100
Sep 23 17:38:01 dist-test-slave-ppjl krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:38:01.396989 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:01.382636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.20.240.66:44359)
0923 17:38:01.382837 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:01.382843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:01.382918 (+ 75us) server_negotiation.cc:407] Connection header received
0923 17:38:01.383098 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:01.383104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:01.383207 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:01.383306 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:01.384546 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.385235 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.386476 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.386876 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.386984 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:01.391187 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:01.391210 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:01.391232 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:01.391288 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:01.394027 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:01.394747 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:01.394757 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:01.394760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:01.394910 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:01.395452 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:01.395458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:01.395462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:01.395847 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:01.396116 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:01.396444 (+ 328us) server_negotiation.cc:299] Negotiation successful
0923 17:38:01.396664 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:38:01.397655 3224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:01.487438 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 3100
I20260923 17:38:01.501698 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 3234
Sep 23 17:38:01 dist-test-slave-ppjl krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:38:01.520298 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:01.502505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.20.240.67:44211)
0923 17:38:01.502811 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:01.502818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:01.502842 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:38:01.502911 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:01.502916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:01.503014 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:01.503138 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:01.504514 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.505785 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.507296 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:01.507999 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:01.508136 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:01.512691 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:01.512710 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:01.512732 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:01.512794 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:01.516794 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:01.517599 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:01.517610 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:01.517614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:01.517820 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:01.518255 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:01.518260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:01.518263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:01.518752 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:01.519073 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:01.519640 (+ 567us) server_negotiation.cc:299] Negotiation successful
0923 17:38:01.519894 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260923 17:38:01.521100 3357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:01.643322 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 3234
I20260923 17:38:01.659050 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 2871
I20260923 17:38:01.660261 2934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:01.774652 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 2871
I20260923 17:38:01.792905 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41471
--webserver_interface=127.20.240.126
--webserver_port=41375
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:41471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:38:02.127826 3411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:02.128146 3411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:02.128293 3411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:02.135798 3411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:38:02.136029 3411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:02.136118 3411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:02.136186 3411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:38:02.136250 3411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:38:02.149202 3411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:41471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=41375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:02.150568 3411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:02.152024 3411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:02.174031 3418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:02.174100 3417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:02.174078 3420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:02.175485 3411 server_base.cc:1034] running on GCE node
I20260923 17:38:02.176414 3411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:02.178086 3411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:02.179282 3411 hybrid_clock.cc:648] HybridClock initialized: now 1790185082179222 us; error 43 us; skew 500 ppm
I20260923 17:38:02.179693 3411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:02 dist-test-slave-ppjl krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185082, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:02.183329 3411 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:38:02.185164 3411 webserver.cc:460] Webserver started at http://127.20.240.126:41375/ using document root <none> and password file <none>
I20260923 17:38:02.185943 3411 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:02.186089 3411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:02.192849 3411 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260923 17:38:02.196219 3427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:02.197827 3411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:38:02.198026 3411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "977113bfb73548ccb31b85199ab722fb"
format_stamp: "Formatted at 2026-09-23 17:37:59 on dist-test-slave-ppjl"
server_key: "adf5462740a427597bc2d0f85c100ea0"
server_key_iv: "e25fff446b76458267e12fc62778a571"
server_key_version: "encryptionkey@0"
I20260923 17:38:02.199276 3411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:02.232539 3411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:02.237054 3411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:02.237344 3411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:02.256492 3411 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:41471
I20260923 17:38:02.256495 3478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:41471 every 8 connection(s)
I20260923 17:38:02.258128 3411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:38:02.265923 3479 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:38:02.266615 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 3411
I20260923 17:38:02.267521 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:46557
--local_ip_for_outbound_sockets=127.20.240.65
--tserver_master_addrs=127.20.240.126:41471
--webserver_port=44015
--webserver_interface=127.20.240.65
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:38:02.269665 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: Bootstrap starting.
I20260923 17:38:02.289469 3479 log.cc:824] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:02.301963 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:38:02.302497 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: Bootstrap complete.
I20260923 17:38:02.311885 3479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } }
I20260923 17:38:02.312758 3479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 977113bfb73548ccb31b85199ab722fb, State: Initialized, Role: FOLLOWER
I20260923 17:38:02.313367 3479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } }
I20260923 17:38:02.313607 3479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:38:02.313726 3479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:38:02.313890 3479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:38:02.317889 3479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } }
I20260923 17:38:02.318455 3479 leader_election.cc:302] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 977113bfb73548ccb31b85199ab722fb; no voters:
I20260923 17:38:02.319089 3479 leader_election.cc:288] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:38:02.319258 3483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:38:02.320408 3483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 2 LEADER]: Becoming Leader. State: Replica: 977113bfb73548ccb31b85199ab722fb, State: Running, Role: LEADER
I20260923 17:38:02.321069 3483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } }
I20260923 17:38:02.321630 3479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:38:02.323587 3485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 977113bfb73548ccb31b85199ab722fb. Latest consensus state: current_term: 2 leader_uuid: "977113bfb73548ccb31b85199ab722fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } } }
I20260923 17:38:02.323974 3485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [sys.catalog]: This master's current role is: LEADER
I20260923 17:38:02.324510 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "977113bfb73548ccb31b85199ab722fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } } }
I20260923 17:38:02.324793 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [sys.catalog]: This master's current role is: LEADER
I20260923 17:38:02.335175 3490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:38:02.346292 3490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:38:02.347837 3490 catalog_manager.cc:1259] Loaded cluster ID: e2d91e9f5126447ab61865d19476d167
I20260923 17:38:02.347945 3490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:38:02.351573 3490 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 17:38:02.364813 3500 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:38:02.368762 3490 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 17:38:02.375926 3490 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:38:02.377261 3490 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: Loaded TSK: 0
I20260923 17:38:02.378436 3490 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:38:02.627980 3481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:02.628247 3481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:02.628362 3481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:02.635720 3481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:02.635833 3481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:02.635975 3481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:38:02.645941 3481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:46557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=44015
--tserver_master_addrs=127.20.240.126:41471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:02.647264 3481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:02.648567 3481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:02.663806 3506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:02.663781 3507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:02.664521 3481 server_base.cc:1034] running on GCE node
W20260923 17:38:02.664655 3509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:02.665329 3481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:02.666332 3481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:02.667541 3481 hybrid_clock.cc:648] HybridClock initialized: now 1790185082667476 us; error 66 us; skew 500 ppm
I20260923 17:38:02.667847 3481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:02 dist-test-slave-ppjl krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185082, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:02.671358 3481 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:38:02.672856 3481 webserver.cc:460] Webserver started at http://127.20.240.65:44015/ using document root <none> and password file <none>
I20260923 17:38:02.673537 3481 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:02.673656 3481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:02.679934 3481 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:38:02.683557 3516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:02.685346 3481 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:38:02.685530 3481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "eacbf1b49d4649a2b8fc10108bb20f7e"
format_stamp: "Formatted at 2026-09-23 17:37:59 on dist-test-slave-ppjl"
server_key: "ece5783e5050a7a1d148b70f6ce50eb3"
server_key_iv: "56cfdbdaa1754e34e008c9874418a002"
server_key_version: "encryptionkey@0"
I20260923 17:38:02.686937 3481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:02.710855 3481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:02.716380 3481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:02.716674 3481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:02.717841 3481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:38:02.719609 3481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:38:02.719734 3481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:02.719844 3481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:38:02.719919 3481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:02.752921 3481 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:46557
I20260923 17:38:02.752972 3628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:46557 every 8 connection(s)
I20260923 17:38:02.754360 3481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:38:02.760484 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 3481
I20260923 17:38:02.761085 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:44359
--local_ip_for_outbound_sockets=127.20.240.66
--tserver_master_addrs=127.20.240.126:41471
--webserver_port=36207
--webserver_interface=127.20.240.66
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:38:02 dist-test-slave-ppjl krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185082, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:38:02.778362 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:02.757540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:37815)
0923 17:38:02.758318 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:02.758356 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:02.760127 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:02.761652 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:02.761663 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:02.762219 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:38:02.762765 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:02.762784 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:02.764481 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:02.764485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:02.765715 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:02.765729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:02.766761 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:02.766769 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:02.767053 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:02.767739 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:38:02.767768 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:38:02.770258 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:38:02.773463 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:02.773472 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:02.773490 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:02.773932 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:02.774453 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:02.774459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:02.774463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:02.774697 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:02.775730 (+ 1033us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:38:02.775739 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:38:02.776111 (+ 372us) client_negotiation.cc:769] Sending connection context
0923 17:38:02.776374 (+ 263us) client_negotiation.cc:240] Negotiation successful
0923 17:38:02.776998 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":318,"threads_started":1}
I20260923 17:38:02.778362 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:02.757400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:37815 (local address 127.20.240.126:41471)
0923 17:38:02.758065 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:02.758081 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:02.758478 (+ 397us) server_negotiation.cc:407] Connection header received
0923 17:38:02.760687 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:02.760700 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:02.761050 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:02.761383 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:02.762994 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:02.764311 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:02.765861 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:02.766529 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:02.766708 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:02.770536 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:02.770545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:02.770560 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:02.770598 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:02.773312 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:02.774073 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:02.774082 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:02.774088 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:02.774277 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:02.774892 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:02.774896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:02.774898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:02.775571 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:02.775782 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:02.776605 (+ 823us) server_negotiation.cc:299] Negotiation successful
0923 17:38:02.777006 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":293,"threads_started":1}
I20260923 17:38:02.781810 3629 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41471
I20260923 17:38:02.782135 3629 heartbeater.cc:464] Registering TS with master...
I20260923 17:38:02.782908 3629 heartbeater.cc:511] Master 127.20.240.126:41471 requested a full tablet report, sending...
I20260923 17:38:02.785411 3444 ts_manager.cc:194] Registered new tserver with Master: eacbf1b49d4649a2b8fc10108bb20f7e (127.20.240.65:46557)
I20260923 17:38:02.787585 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:37815
W20260923 17:38:03.087958 3633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:03.088259 3633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:03.088356 3633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:03.096094 3633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:03.096240 3633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:03.096365 3633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:38:03.106897 3633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:44359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=36207
--tserver_master_addrs=127.20.240.126:41471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:03.108217 3633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:03.109495 3633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:03.122551 3641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:03.122610 3638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:03.123315 3639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:03.123566 3633 server_base.cc:1034] running on GCE node
I20260923 17:38:03.124234 3633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:03.125247 3633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:03.126475 3633 hybrid_clock.cc:648] HybridClock initialized: now 1790185083126417 us; error 50 us; skew 500 ppm
I20260923 17:38:03.126845 3633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:03 dist-test-slave-ppjl krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185083, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:03.130093 3633 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:38:03.131635 3633 webserver.cc:460] Webserver started at http://127.20.240.66:36207/ using document root <none> and password file <none>
I20260923 17:38:03.132340 3633 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:03.132470 3633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:03.138002 3633 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:38:03.140785 3648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:03.142355 3633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:38:03.142529 3633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "a255d4034a1247838478b91846b78652"
format_stamp: "Formatted at 2026-09-23 17:38:00 on dist-test-slave-ppjl"
server_key: "0e22905ad33450bdcdde0bf612ca2a2d"
server_key_iv: "e4d8ca6a13c3ea27ade1aeee27650b78"
server_key_version: "encryptionkey@0"
I20260923 17:38:03.143647 3633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:03.188437 3633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:03.192852 3633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:03.193122 3633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:03.194095 3633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:38:03.195492 3633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:38:03.195588 3633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:03.195690 3633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:38:03.195765 3633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:03.227197 3633 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:44359
I20260923 17:38:03.227276 3760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:44359 every 8 connection(s)
I20260923 17:38:03.228688 3633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:38:03.234241 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 3633
I20260923 17:38:03.234959 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:44211
--local_ip_for_outbound_sockets=127.20.240.67
--tserver_master_addrs=127.20.240.126:41471
--webserver_port=38179
--webserver_interface=127.20.240.67
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:38:03 dist-test-slave-ppjl krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185083, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:38:03.250207 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.231339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:42779 (local address 127.20.240.126:41471)
0923 17:38:03.231516 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:03.231520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:03.232793 (+ 1273us) server_negotiation.cc:407] Connection header received
0923 17:38:03.234704 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:03.234710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:03.234810 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:03.234935 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:03.236684 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.237667 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.239635 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.240142 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.240222 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.244394 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:03.244400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:03.244403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:03.244443 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:03.246986 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:03.247805 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:03.247810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:03.247812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:03.247999 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:03.248447 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:03.248453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:03.248456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:03.248745 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:03.248889 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:03.249613 (+ 724us) server_negotiation.cc:299] Negotiation successful
0923 17:38:03.249881 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:38:03.251278 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.231801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:42779)
0923 17:38:03.232610 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:03.232655 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:03.234379 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:03.235219 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:03.235232 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:03.235859 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:38:03.236467 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:03.236486 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.237844 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.237848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:03.239466 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:03.239481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.240525 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.240530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:03.240784 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.241507 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:38:03.241534 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:38:03.244135 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:38:03.247129 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:03.247139 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:03.247194 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:03.247665 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:03.248116 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:03.248121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:03.248125 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:03.248325 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:03.248874 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:38:03.248882 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:38:03.249239 (+ 357us) client_negotiation.cc:769] Sending connection context
0923 17:38:03.249481 (+ 242us) client_negotiation.cc:240] Negotiation successful
0923 17:38:03.250073 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":295,"threads_started":1}
I20260923 17:38:03.252982 3761 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41471
I20260923 17:38:03.253314 3761 heartbeater.cc:464] Registering TS with master...
I20260923 17:38:03.254073 3761 heartbeater.cc:511] Master 127.20.240.126:41471 requested a full tablet report, sending...
I20260923 17:38:03.255581 3444 ts_manager.cc:194] Registered new tserver with Master: a255d4034a1247838478b91846b78652 (127.20.240.66:44359)
I20260923 17:38:03.256525 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:42779
W20260923 17:38:03.576551 3764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:03.576939 3764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:03.577055 3764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:03.584575 3764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:03.584697 3764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:03.584833 3764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:38:03.594767 3764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:44211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=38179
--tserver_master_addrs=127.20.240.126:41471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:03.596088 3764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:03.597409 3764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:03.610656 3770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:03.610724 3769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:03.610724 3772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:03.611531 3764 server_base.cc:1034] running on GCE node
I20260923 17:38:03.612243 3764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:03.613251 3764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:03.614483 3764 hybrid_clock.cc:648] HybridClock initialized: now 1790185083614438 us; error 35 us; skew 500 ppm
I20260923 17:38:03.614864 3764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:03 dist-test-slave-ppjl krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185083, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:03.618441 3764 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:38:03.620033 3764 webserver.cc:460] Webserver started at http://127.20.240.67:38179/ using document root <none> and password file <none>
I20260923 17:38:03.620704 3764 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:03.620843 3764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:03.626632 3764 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:38:03.629702 3779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:03.631356 3764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:38:03.631526 3764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "93a1ff7e14d64d418e9b4751ac279532"
format_stamp: "Formatted at 2026-09-23 17:38:00 on dist-test-slave-ppjl"
server_key: "7ea654838722b2a081ed3059704990d4"
server_key_iv: "53a169c046b5eee3a7d3df0f56a229fb"
server_key_version: "encryptionkey@0"
I20260923 17:38:03.632597 3764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:03.656602 3764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:03.661087 3764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:03.661336 3764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:03.662341 3764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:38:03.663710 3764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:38:03.663806 3764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:03.663884 3764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:38:03.663936 3764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:03.695456 3764 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:44211
I20260923 17:38:03.695514 3891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:44211 every 8 connection(s)
I20260923 17:38:03.696899 3764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:38:03.699928 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 3764
Sep 23 17:38:03 dist-test-slave-ppjl krb5kdc[2858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185083, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:38:03.721143 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.699450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:45123 (local address 127.20.240.126:41471)
0923 17:38:03.699664 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:03.699671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:03.700995 (+ 1324us) server_negotiation.cc:407] Connection header received
0923 17:38:03.704005 (+ 3010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:03.704012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:03.704114 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:03.704188 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:03.706825 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.707952 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.710158 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.710827 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.710967 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.714729 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:03.714734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:03.714737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:03.714769 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:03.717559 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:03.718490 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:03.718494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:03.718497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:03.718637 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:03.719149 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:03.719180 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:03.719184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:03.719563 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:03.719727 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:03.720621 (+ 894us) server_negotiation.cc:299] Negotiation successful
0923 17:38:03.720792 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:38:03.722219 3895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.700867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.20.240.126:41471)
0923 17:38:03.707535 (+ 6668us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:03.707541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:03.707560 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:38:03.707610 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:03.707615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:03.707703 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:03.707823 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:03.709708 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.710868 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.712723 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.713184 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.713272 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.715948 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:03.715952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:03.715956 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:03.716000 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:03.719028 (+ 3028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:03.719994 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:03.720000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:03.720002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:03.720143 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:03.720516 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:03.720521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:03.720524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:03.720866 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:03.721118 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:03.721620 (+ 502us) server_negotiation.cc:299] Negotiation successful
0923 17:38:03.721864 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6549,"thread_start_us":6423,"threads_started":1}
I20260923 17:38:03.722733 3894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.700012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:45123)
0923 17:38:03.700832 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:03.700883 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:03.703623 (+ 2740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:03.704786 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:03.704799 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:03.705649 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:38:03.706569 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:03.706597 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.708130 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.708136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:03.710020 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:03.710034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.711071 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.711079 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:03.711405 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.712098 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:38:03.712128 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:38:03.714514 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:38:03.717711 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:03.717722 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:03.717742 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:03.718368 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:03.718746 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:03.718751 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:03.718756 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:03.719018 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:03.719711 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:38:03.719717 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:38:03.720170 (+ 453us) client_negotiation.cc:769] Sending connection context
0923 17:38:03.720533 (+ 363us) client_negotiation.cc:240] Negotiation successful
0923 17:38:03.721118 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":290,"threads_started":1}
I20260923 17:38:03.724260 3892 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41471
I20260923 17:38:03.724570 3892 heartbeater.cc:464] Registering TS with master...
I20260923 17:38:03.725375 3892 heartbeater.cc:511] Master 127.20.240.126:41471 requested a full tablet report, sending...
I20260923 17:38:03.726716 3444 ts_manager.cc:194] Registered new tserver with Master: 93a1ff7e14d64d418e9b4751ac279532 (127.20.240.67:44211)
I20260923 17:38:03.727612 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:45123
I20260923 17:38:03.737641 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:38:03.744545 3895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.738533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.20.240.126:41471)
0923 17:38:03.738682 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:03.738686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:03.739205 (+ 519us) server_negotiation.cc:407] Connection header received
0923 17:38:03.739519 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:03.739524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:03.739667 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:03.739831 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:38:03.740304 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.741041 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.742365 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.742929 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.743026 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.743569 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:38:03.744091 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:38:03.744204 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:03.744246 (+ 42us) server_negotiation.cc:299] Negotiation successful
0923 17:38:03.744348 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 17:38:03.745512 3444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:38:03.752831 3444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:38:03.785073 3902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.764904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46557 (local address 127.0.0.1:59848)
0923 17:38:03.765842 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:03.765893 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:03.766087 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:03.777972 (+ 11885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:03.777983 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:03.778032 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:03.778535 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:03.778553 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.780556 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.780561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:03.782659 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:03.782673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.784046 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.784052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:03.784299 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.784345 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 17:38:03.784470 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 17:38:03.784669 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":26,"thread_start_us":716,"threads_started":1}
I20260923 17:38:03.786072 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.765046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.20.240.65:46557)
0923 17:38:03.777082 (+ 12036us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:03.777092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:03.777128 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:38:03.777261 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:03.777272 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:03.777588 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:03.778764 (+ 1176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:03.779040 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.780422 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.782814 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.783818 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.783944 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.784197 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:03.785580 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0923 17:38:03.785760 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11849,"thread_start_us":11717,"threads_started":1}
I20260923 17:38:03.787242 3906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.765271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:44211 (local address 127.0.0.1:57338)
0923 17:38:03.777896 (+ 12625us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:03.777919 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:03.778076 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:03.779249 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:03.779252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:03.779273 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:03.779639 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:03.779651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.781618 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.781622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:03.784309 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:03.784325 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.786489 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.786494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:03.786607 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.786630 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:38:03.786702 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 17:38:03.786838 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12492,"mutex_wait_us":163,"thread_start_us":12726,"threads_started":1}
I20260923 17:38:03.787315 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.765334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57338 (local address 127.20.240.67:44211)
0923 17:38:03.776314 (+ 10980us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:03.776327 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:03.778238 (+ 1911us) server_negotiation.cc:407] Connection header received
0923 17:38:03.778359 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:03.778367 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:03.778620 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:03.778799 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:03.779832 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.781458 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.784484 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.785998 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.786129 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.786347 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:03.786797 (+ 450us) server_negotiation.cc:299] Negotiation successful
0923 17:38:03.786963 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10747,"thread_start_us":11009,"threads_started":1}
I20260923 17:38:03.787762 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.766466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:44359 (local address 127.0.0.1:53584)
0923 17:38:03.778270 (+ 11804us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:03.778287 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:03.778400 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:03.779258 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:03.779263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:03.779288 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:03.779755 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:03.779769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.781615 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.781620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:03.784393 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:03.784404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.786145 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.786151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:03.786337 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.787187 (+ 850us) client_negotiation.cc:769] Sending connection context
0923 17:38:03.787275 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 17:38:03.787419 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11669,"mutex_wait_us":119,"thread_start_us":11155,"threads_started":1}
I20260923 17:38:03.788268 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.766958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.20.240.66:44359)
0923 17:38:03.777407 (+ 10449us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:03.777417 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:03.778521 (+ 1104us) server_negotiation.cc:407] Connection header received
0923 17:38:03.778618 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:03.778624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:03.778839 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:03.779000 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:03.779919 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.781478 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.784539 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.785674 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.785797 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.785984 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:03.787787 (+ 1803us) server_negotiation.cc:299] Negotiation successful
0923 17:38:03.787956 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10277,"thread_start_us":10165,"threads_started":1}
I20260923 17:38:03.789804 3564 tablet_service.cc:1467] Processing CreateTablet for tablet 8a4c03cf4faa4054b06c33852596ad3c (DEFAULT_TABLE table=test-table [id=9e43ae199af149fd96134eb32b887a3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:38:03.790889 3629 heartbeater.cc:503] Master 127.20.240.126:41471 was elected leader, sending a full tablet report...
I20260923 17:38:03.790603 3827 tablet_service.cc:1467] Processing CreateTablet for tablet 8a4c03cf4faa4054b06c33852596ad3c (DEFAULT_TABLE table=test-table [id=9e43ae199af149fd96134eb32b887a3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:38:03.791311 3564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a4c03cf4faa4054b06c33852596ad3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:38:03.791956 3827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a4c03cf4faa4054b06c33852596ad3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:38:03.794570 3696 tablet_service.cc:1467] Processing CreateTablet for tablet 8a4c03cf4faa4054b06c33852596ad3c (DEFAULT_TABLE table=test-table [id=9e43ae199af149fd96134eb32b887a3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:38:03.795899 3696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a4c03cf4faa4054b06c33852596ad3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:38:03.803893 3908 tablet_bootstrap.cc:492] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e: Bootstrap starting.
I20260923 17:38:03.807210 3908 tablet_bootstrap.cc:654] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e: Neither blocks nor log segments found. Creating new log.
I20260923 17:38:03.807969 3909 tablet_bootstrap.cc:492] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532: Bootstrap starting.
I20260923 17:38:03.809624 3908 log.cc:824] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:03.811651 3909 tablet_bootstrap.cc:654] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532: Neither blocks nor log segments found. Creating new log.
I20260923 17:38:03.812937 3909 log.cc:824] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:03.813035 3908 tablet_bootstrap.cc:492] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e: No bootstrap required, opened a new log
I20260923 17:38:03.813176 3911 tablet_bootstrap.cc:492] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652: Bootstrap starting.
I20260923 17:38:03.813336 3908 ts_tablet_manager.cc:1397] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 17:38:03.816495 3911 tablet_bootstrap.cc:654] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652: Neither blocks nor log segments found. Creating new log.
I20260923 17:38:03.817703 3911 log.cc:824] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:03.818140 3909 tablet_bootstrap.cc:492] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532: No bootstrap required, opened a new log
I20260923 17:38:03.818459 3909 ts_tablet_manager.cc:1397] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260923 17:38:03.824049 3911 tablet_bootstrap.cc:492] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652: No bootstrap required, opened a new log
I20260923 17:38:03.824388 3911 ts_tablet_manager.cc:1397] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260923 17:38:03.824241 3908 raft_consensus.cc:348] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } } peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } }
I20260923 17:38:03.824738 3908 raft_consensus.cc:374] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:38:03.824884 3908 raft_consensus.cc:729] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eacbf1b49d4649a2b8fc10108bb20f7e, State: Initialized, Role: FOLLOWER
I20260923 17:38:03.825635 3908 consensus_queue.cc:260] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } } peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } }
I20260923 17:38:03.827594 3908 ts_tablet_manager.cc:1428] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260923 17:38:03.829149 3909 raft_consensus.cc:348] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } } peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } }
I20260923 17:38:03.829654 3909 raft_consensus.cc:374] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:38:03.829840 3909 raft_consensus.cc:729] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a1ff7e14d64d418e9b4751ac279532, State: Initialized, Role: FOLLOWER
I20260923 17:38:03.831899 3909 consensus_queue.cc:260] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } } peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } }
I20260923 17:38:03.833396 3892 heartbeater.cc:503] Master 127.20.240.126:41471 was elected leader, sending a full tablet report...
I20260923 17:38:03.833240 3911 raft_consensus.cc:348] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } } peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } }
I20260923 17:38:03.833703 3911 raft_consensus.cc:374] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:38:03.833799 3911 raft_consensus.cc:729] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a255d4034a1247838478b91846b78652, State: Initialized, Role: FOLLOWER
I20260923 17:38:03.833894 3909 ts_tablet_manager.cc:1428] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260923 17:38:03.834409 3911 consensus_queue.cc:260] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } } peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } }
I20260923 17:38:03.835611 3761 heartbeater.cc:503] Master 127.20.240.126:41471 was elected leader, sending a full tablet report...
I20260923 17:38:03.836215 3911 ts_tablet_manager.cc:1428] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
W20260923 17:38:03.948505 3893 tablet.cc:2369] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:38:03.968215 3916 raft_consensus.cc:482] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:38:03.968534 3916 raft_consensus.cc:504] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } } peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } }
I20260923 17:38:03.970319 3916 leader_election.cc:288] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eacbf1b49d4649a2b8fc10108bb20f7e (127.20.240.65:46557), 93a1ff7e14d64d418e9b4751ac279532 (127.20.240.67:44211)
I20260923 17:38:03.977152 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.970799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:51477 (local address 127.20.240.65:46557)
0923 17:38:03.971007 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:03.971013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:03.971692 (+ 679us) server_negotiation.cc:407] Connection header received
0923 17:38:03.972017 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:03.972022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:03.972102 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:03.972163 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:03.972878 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.973607 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.975412 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.976176 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.976345 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.976426 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:03.976731 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 17:38:03.976903 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:38:03.977279 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.970863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:41141 (local address 127.20.240.67:44211)
0923 17:38:03.971049 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:03.971054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:03.972418 (+ 1364us) server_negotiation.cc:407] Connection header received
0923 17:38:03.972587 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:03.972590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:03.972648 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:03.972721 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:03.973244 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.973927 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.975571 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.976345 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.976495 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.976568 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:03.976920 (+ 352us) server_negotiation.cc:299] Negotiation successful
0923 17:38:03.977042 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:38:03.977306 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.970798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46557 (local address 127.20.240.66:51477)
0923 17:38:03.971590 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:03.971608 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:03.971827 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:03.972372 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:03.972375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:03.972397 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:03.972676 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:03.972685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.973738 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.973743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:03.975259 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:03.975272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.976394 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.976399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:03.976580 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.976605 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:38:03.976691 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 17:38:03.976837 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":3,"thread_start_us":406,"threads_started":1}
I20260923 17:38:03.977397 3918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:03.971329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:44211 (local address 127.20.240.66:41141)
0923 17:38:03.972302 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:03.972318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:03.972454 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:03.972750 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:03.972754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:03.972776 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:03.973096 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:03.973105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.974049 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.974054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:03.975432 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:03.975444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:03.976540 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:03.976547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:03.976745 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:03.976768 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:38:03.976852 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:38:03.976989 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"mutex_wait_us":43,"thread_start_us":649,"threads_started":1}
I20260923 17:38:03.978421 3847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a4c03cf4faa4054b06c33852596ad3c" candidate_uuid: "a255d4034a1247838478b91846b78652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a1ff7e14d64d418e9b4751ac279532" is_pre_election: true
I20260923 17:38:03.978421 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a4c03cf4faa4054b06c33852596ad3c" candidate_uuid: "a255d4034a1247838478b91846b78652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" is_pre_election: true
I20260923 17:38:03.979018 3847 raft_consensus.cc:2454] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a255d4034a1247838478b91846b78652 in term 0.
I20260923 17:38:03.979018 3584 raft_consensus.cc:2454] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a255d4034a1247838478b91846b78652 in term 0.
I20260923 17:38:03.979784 3652 leader_election.cc:302] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 93a1ff7e14d64d418e9b4751ac279532, a255d4034a1247838478b91846b78652, eacbf1b49d4649a2b8fc10108bb20f7e; no voters:
I20260923 17:38:03.980185 3916 raft_consensus.cc:2781] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:38:03.980389 3916 raft_consensus.cc:482] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:38:03.980527 3916 raft_consensus.cc:3037] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [term 0 FOLLOWER]: Advancing to term 1
W20260923 17:38:03.980594 3762 tablet.cc:2369] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:38:03.982908 3916 raft_consensus.cc:504] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } } peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } }
I20260923 17:38:03.983523 3916 leader_election.cc:288] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [CANDIDATE]: Term 1 election: Requested vote from peers eacbf1b49d4649a2b8fc10108bb20f7e (127.20.240.65:46557), 93a1ff7e14d64d418e9b4751ac279532 (127.20.240.67:44211)
I20260923 17:38:03.983994 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a4c03cf4faa4054b06c33852596ad3c" candidate_uuid: "a255d4034a1247838478b91846b78652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e"
I20260923 17:38:03.984037 3847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a4c03cf4faa4054b06c33852596ad3c" candidate_uuid: "a255d4034a1247838478b91846b78652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a1ff7e14d64d418e9b4751ac279532"
I20260923 17:38:03.984175 3584 raft_consensus.cc:3037] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:38:03.984217 3847 raft_consensus.cc:3037] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:38:03.986364 3584 raft_consensus.cc:2454] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a255d4034a1247838478b91846b78652 in term 1.
I20260923 17:38:03.986500 3847 raft_consensus.cc:2454] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a255d4034a1247838478b91846b78652 in term 1.
I20260923 17:38:03.986806 3652 leader_election.cc:302] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a255d4034a1247838478b91846b78652, eacbf1b49d4649a2b8fc10108bb20f7e; no voters:
I20260923 17:38:03.987119 3916 raft_consensus.cc:2781] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:38:03.987550 3916 raft_consensus.cc:686] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [term 1 LEADER]: Becoming Leader. State: Replica: a255d4034a1247838478b91846b78652, State: Running, Role: LEADER
I20260923 17:38:03.988020 3916 consensus_queue.cc:237] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } } peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } }
I20260923 17:38:03.992930 3443 catalog_manager.cc:5495] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 reported cstate change: term changed from 0 to 1, leader changed from <none> to a255d4034a1247838478b91846b78652 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "a255d4034a1247838478b91846b78652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a255d4034a1247838478b91846b78652" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44359 } health_report { overall_health: HEALTHY } } }
W20260923 17:38:04.006735 3630 tablet.cc:2369] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:38:04.031279 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:04.025887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53590 (local address 127.20.240.66:44359)
0923 17:38:04.026074 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:04.026079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:04.026149 (+ 70us) server_negotiation.cc:407] Connection header received
0923 17:38:04.026347 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:04.026352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:04.026436 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:04.026566 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:38:04.026890 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:04.027667 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:04.029219 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:04.029636 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:04.029740 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:04.030215 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:38:04.030734 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:38:04.030902 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:04.030947 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 17:38:04.031019 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:38:04.040233 3847 raft_consensus.cc:1261] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532 [term 1 FOLLOWER]: Refusing update from remote peer a255d4034a1247838478b91846b78652: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:38:04.040233 3584 raft_consensus.cc:1261] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e [term 1 FOLLOWER]: Refusing update from remote peer a255d4034a1247838478b91846b78652: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:38:04.041363 3919 consensus_queue.cc:1035] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93a1ff7e14d64d418e9b4751ac279532" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:38:04.041761 3916 consensus_queue.cc:1035] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eacbf1b49d4649a2b8fc10108bb20f7e" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:38:04.071518 3443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:38:04.071796 3443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:38:04.075328 3443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: Sending DeleteTablet for 3 replicas of tablet 8a4c03cf4faa4054b06c33852596ad3c
I20260923 17:38:04.076602 3827 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a4c03cf4faa4054b06c33852596ad3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:38:04 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:57338
I20260923 17:38:04.076637 3696 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a4c03cf4faa4054b06c33852596ad3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:38:04 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:53584
I20260923 17:38:04.076763 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 3481
I20260923 17:38:04.078056 3564 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a4c03cf4faa4054b06c33852596ad3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:38:04 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:59848
I20260923 17:38:04.088579 3933 tablet_replica.cc:330] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e: stopping tablet replica
I20260923 17:38:04.089193 3931 tablet_replica.cc:330] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652: stopping tablet replica
I20260923 17:38:04.089287 3933 raft_consensus.cc:2229] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:38:04.089844 3932 tablet_replica.cc:330] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532: stopping tablet replica
I20260923 17:38:04.090020 3933 raft_consensus.cc:2258] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:38:04.090456 3932 raft_consensus.cc:2229] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:38:04.091562 3931 raft_consensus.cc:2229] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:38:04.092155 3932 raft_consensus.cc:2258] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:38:04.093104 3931 raft_consensus.cc:2258] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:38:04.093384 3933 ts_tablet_manager.cc:1905] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:38:04.095487 3931 ts_tablet_manager.cc:1905] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:38:04.097342 3932 ts_tablet_manager.cc:1905] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:38:04.100106 3903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:04.079627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59854 (local address 127.20.240.65:46557)
0923 17:38:04.079841 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:04.079847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:04.079880 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:38:04.079948 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:04.079953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:04.080040 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:04.080154 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:04.082324 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:04.083678 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:04.085950 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:04.086708 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:04.086838 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:04.090870 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:04.090883 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:04.090900 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:04.090946 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:04.095139 (+ 4193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:04.096445 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:04.096455 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:04.096459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:04.096637 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:04.097198 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:04.097203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:04.097206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:04.097705 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:04.098040 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:04.099441 (+ 1401us) server_negotiation.cc:299] Negotiation successful
0923 17:38:04.099692 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:38:04.102918 3624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:04.102923 3931 ts_tablet_manager.cc:1918] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:38:04.103106 3931 log.cc:1196] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/8a4c03cf4faa4054b06c33852596ad3c
I20260923 17:38:04.103788 3931 ts_tablet_manager.cc:1939] T 8a4c03cf4faa4054b06c33852596ad3c P a255d4034a1247838478b91846b78652: Deleting consensus metadata
I20260923 17:38:04.104413 3932 ts_tablet_manager.cc:1918] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:38:04.104624 3932 log.cc:1196] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/8a4c03cf4faa4054b06c33852596ad3c
I20260923 17:38:04.105212 3932 ts_tablet_manager.cc:1939] T 8a4c03cf4faa4054b06c33852596ad3c P 93a1ff7e14d64d418e9b4751ac279532: Deleting consensus metadata
I20260923 17:38:04.105971 3431 catalog_manager.cc:4841] TS a255d4034a1247838478b91846b78652 (127.20.240.66:44359): tablet 8a4c03cf4faa4054b06c33852596ad3c (table test-table [id=9e43ae199af149fd96134eb32b887a3e]) successfully deleted
I20260923 17:38:04.106971 3429 catalog_manager.cc:4841] TS 93a1ff7e14d64d418e9b4751ac279532 (127.20.240.67:44211): tablet 8a4c03cf4faa4054b06c33852596ad3c (table test-table [id=9e43ae199af149fd96134eb32b887a3e]) successfully deleted
I20260923 17:38:04.201612 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 3481
W20260923 17:38:04.216135 3431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.240.65:46557 (error 108)
I20260923 17:38:04.216848 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 3633
I20260923 17:38:04.218348 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:04.217920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46557 (local address 127.0.0.1:59866)
0923 17:38:04.218087 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:04.218187 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 17:38:04.218868 3431 catalog_manager.cc:4584] TS eacbf1b49d4649a2b8fc10108bb20f7e (127.20.240.65:46557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a4c03cf4faa4054b06c33852596ad3c: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46557: connect: Connection refused (error 111)
I20260923 17:38:04.232904 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:04.217689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53606 (local address 127.20.240.66:44359)
0923 17:38:04.217932 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:04.217938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:04.218199 (+ 261us) server_negotiation.cc:407] Connection header received
0923 17:38:04.218381 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:04.218387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:04.218478 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:04.218580 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:04.220115 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:04.220883 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:04.222552 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:04.222986 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:04.223104 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:04.225923 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:04.225937 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:04.225952 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:04.225985 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:04.229750 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:04.230629 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:04.230637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:04.230640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:04.230812 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:04.231252 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:04.231257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:04.231260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:04.231686 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:04.232017 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:04.232309 (+ 292us) server_negotiation.cc:299] Negotiation successful
0923 17:38:04.232533 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 17:38:04.233583 3756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:04.272467 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:04.271972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46557 (local address 127.0.0.1:59880)
0923 17:38:04.272171 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:04.272316 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 17:38:04.273478 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:04.273131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46557 (local address 127.0.0.1:59886)
0923 17:38:04.273288 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:04.273369 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 17:38:04.334295 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:04.333773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46557 (local address 127.0.0.1:59892)
0923 17:38:04.334015 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:04.334116 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260923 17:38:04.335455 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:04.335051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46557 (local address 127.0.0.1:59902)
0923 17:38:04.335253 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:04.335333 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 17:38:04.348483 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 3633
I20260923 17:38:04.364544 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 3764
I20260923 17:38:04.377991 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:04.365341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.20.240.67:44211)
0923 17:38:04.365532 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:04.365537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:04.365555 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:38:04.365636 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:04.365639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:04.365719 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:04.365799 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:04.367249 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:04.368017 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:04.369440 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:04.369888 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:04.369996 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:04.372614 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:04.372629 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:04.372646 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:04.372682 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:04.375313 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:04.375993 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:04.376001 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:04.376003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:04.376155 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:04.376507 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:04.376512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:04.376516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:04.376866 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:04.377155 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:04.377439 (+ 284us) server_negotiation.cc:299] Negotiation successful
0923 17:38:04.377668 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:38:04.378572 3887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:04.416234 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:04.415646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46557 (local address 127.0.0.1:59912)
0923 17:38:04.415921 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:04.416035 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260923 17:38:04.417346 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:04.416947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46557 (local address 127.0.0.1:59920)
0923 17:38:04.417145 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:04.417227 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260923 17:38:04.466558 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 3764
I20260923 17:38:04.479697 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 3411
I20260923 17:38:04.480892 3474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:04.594036 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:04.593164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:46557 (local address 127.0.0.1:59934)
0923 17:38:04.593395 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:04.593529 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:46557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 17:38:04.594089 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 3411
I20260923 17:38:04.612214 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41471
--webserver_interface=127.20.240.126
--webserver_port=41375
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:41471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:38:04.945175 3939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:04.945482 3939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:04.945616 3939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:04.953326 3939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:38:04.953464 3939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:04.953518 3939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:04.953603 3939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:38:04.953658 3939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:38:04.964102 3939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:41471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=41375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:04.965598 3939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:04.966948 3939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:04.978916 3944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:04.978932 3945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:04.979105 3947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:04.979954 3939 server_base.cc:1034] running on GCE node
I20260923 17:38:04.980808 3939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:04.982357 3939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:04.983565 3939 hybrid_clock.cc:648] HybridClock initialized: now 1790185084983517 us; error 36 us; skew 500 ppm
I20260923 17:38:04.983951 3939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:04 dist-test-slave-ppjl krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185084, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:04.991091 3939 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:38:04.992894 3939 webserver.cc:460] Webserver started at http://127.20.240.126:41375/ using document root <none> and password file <none>
I20260923 17:38:04.993654 3939 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:04.993782 3939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:04.999591 3939 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:38:05.002523 3954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:05.004070 3939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:38:05.004238 3939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "977113bfb73548ccb31b85199ab722fb"
format_stamp: "Formatted at 2026-09-23 17:37:59 on dist-test-slave-ppjl"
server_key: "adf5462740a427597bc2d0f85c100ea0"
server_key_iv: "e25fff446b76458267e12fc62778a571"
server_key_version: "encryptionkey@0"
I20260923 17:38:05.005286 3939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:05.021934 3939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:05.026341 3939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:05.026607 3939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:05.045413 3939 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:41471
I20260923 17:38:05.045435 4005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:41471 every 8 connection(s)
I20260923 17:38:05.046944 3939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:38:05.054229 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 3939
I20260923 17:38:05.054265 4006 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:38:05.055066 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:46557
--local_ip_for_outbound_sockets=127.20.240.65
--tserver_master_addrs=127.20.240.126:41471
--webserver_port=44015
--webserver_interface=127.20.240.65
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:38:05.057888 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: Bootstrap starting.
I20260923 17:38:05.078333 4006 log.cc:824] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:05.095036 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:38:05.095613 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb: Bootstrap complete.
I20260923 17:38:05.104846 4006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } }
I20260923 17:38:05.105767 4006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 977113bfb73548ccb31b85199ab722fb, State: Initialized, Role: FOLLOWER
I20260923 17:38:05.106379 4006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } }
I20260923 17:38:05.106603 4006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:38:05.106714 4006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:38:05.106889 4006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 2 FOLLOWER]: Advancing to term 3
I20260923 17:38:05.110939 4006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } }
I20260923 17:38:05.111478 4006 leader_election.cc:302] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 977113bfb73548ccb31b85199ab722fb; no voters:
I20260923 17:38:05.112089 4006 leader_election.cc:288] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 17:38:05.112234 4010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 3 FOLLOWER]: Leader election won for term 3
I20260923 17:38:05.114181 4006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:38:05.115442 4010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [term 3 LEADER]: Becoming Leader. State: Replica: 977113bfb73548ccb31b85199ab722fb, State: Running, Role: LEADER
I20260923 17:38:05.116123 4010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } }
I20260923 17:38:05.118719 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 977113bfb73548ccb31b85199ab722fb. Latest consensus state: current_term: 3 leader_uuid: "977113bfb73548ccb31b85199ab722fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } } }
I20260923 17:38:05.119129 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [sys.catalog]: This master's current role is: LEADER
I20260923 17:38:05.119722 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "977113bfb73548ccb31b85199ab722fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977113bfb73548ccb31b85199ab722fb" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41471 } } }
I20260923 17:38:05.119982 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977113bfb73548ccb31b85199ab722fb [sys.catalog]: This master's current role is: LEADER
I20260923 17:38:05.123900 4017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:38:05.132547 4017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:38:05.134115 4017 catalog_manager.cc:1259] Loaded cluster ID: e2d91e9f5126447ab61865d19476d167
I20260923 17:38:05.134287 4017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 17:38:05.136572 4017 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f540aecf9ad google::LogMessage::Fail() at ??:0
@ 0x7f540aed288a google::LogMessage::SendToLog() at ??:0
@ 0x7f540aecf4de google::LogMessage::Flush() at ??:0
@ 0x7f540aed0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5420902512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f54208ff976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5420987610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f5420987439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f540db1b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f540db1e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f540db1e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f540daf9eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5410dd36db start_thread at ??:0
@ 0x7f5409d6961f clone at ??:0
W20260923 17:38:05.421444 4008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:05.421880 4008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:05.422030 4008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:05.433254 4008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:05.433429 4008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:05.433589 4008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:38:05.445545 4008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:46557
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=44015
--tserver_master_addrs=127.20.240.126:41471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:05.446831 4008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:05.448216 4008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:05.461696 4049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:05.461784 4051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:05.461745 4048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:05.463007 4008 server_base.cc:1034] running on GCE node
I20260923 17:38:05.463752 4008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:05.464891 4008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:05.466126 4008 hybrid_clock.cc:648] HybridClock initialized: now 1790185085466061 us; error 56 us; skew 500 ppm
I20260923 17:38:05.466485 4008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:05 dist-test-slave-ppjl krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185085, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:05.469892 4008 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:38:05.471587 4008 webserver.cc:460] Webserver started at http://127.20.240.65:44015/ using document root <none> and password file <none>
I20260923 17:38:05.472313 4008 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:05.472426 4008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:05.478205 4008 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:38:05.481525 4058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:05.483315 4008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:38:05.483494 4008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "eacbf1b49d4649a2b8fc10108bb20f7e"
format_stamp: "Formatted at 2026-09-23 17:37:59 on dist-test-slave-ppjl"
server_key: "ece5783e5050a7a1d148b70f6ce50eb3"
server_key_iv: "56cfdbdaa1754e34e008c9874418a002"
server_key_version: "encryptionkey@0"
I20260923 17:38:05.484508 4008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:05.499701 4008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:05.504405 4008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:05.504647 4008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:05.505626 4008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:38:05.507743 4065 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260923 17:38:05.512969 4065 ts_tablet_manager.cc:1852] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260923 17:38:05.513957 4065 ts_tablet_manager.cc:1905] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:38:05.521339 4065 ts_tablet_manager.cc:1918] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:38:05.521533 4065 log.cc:1196] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/8a4c03cf4faa4054b06c33852596ad3c
I20260923 17:38:05.521962 4065 ts_tablet_manager.cc:1939] T 8a4c03cf4faa4054b06c33852596ad3c P eacbf1b49d4649a2b8fc10108bb20f7e: Deleting consensus metadata
I20260923 17:38:05.522384 4008 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260923 17:38:05.522518 4008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.000s
I20260923 17:38:05.522650 4008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:38:05.522719 4008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:05.555110 4008 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:46557
I20260923 17:38:05.555181 4171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:46557 every 8 connection(s)
I20260923 17:38:05.556515 4008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:38:05.558516 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 4008
I20260923 17:38:05.559116 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:44359
--local_ip_for_outbound_sockets=127.20.240.66
--tserver_master_addrs=127.20.240.126:41471
--webserver_port=36207
--webserver_interface=127.20.240.66
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:38:05.561939 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:05.559576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:57163)
0923 17:38:05.560432 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:05.560733 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":301,"threads_started":1}
W20260923 17:38:05.562695 4172 heartbeater.cc:644] Failed to heartbeat to 127.20.240.126:41471 (0 consecutive failures): Network error: Failed to ping master at 127.20.240.126:41471: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
I20260923 17:38:05.563707 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:05.563285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:44611)
0923 17:38:05.563482 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:05.563571 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 17:38:05.564698 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:05.564343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:58041)
0923 17:38:05.564480 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:05.564564 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 17:38:05.564915 4172 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:38:05.890448 4175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:05.890743 4175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:05.890894 4175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:05.898298 4175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:05.898420 4175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:05.898571 4175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:38:05.908694 4175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:44359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=36207
--tserver_master_addrs=127.20.240.126:41471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:05.910032 4175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:05.911391 4175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:05.924858 4181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:05.924780 4183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:05.924949 4180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:05.925444 4175 server_base.cc:1034] running on GCE node
I20260923 17:38:05.926209 4175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:05.927254 4175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:05.928489 4175 hybrid_clock.cc:648] HybridClock initialized: now 1790185085928423 us; error 55 us; skew 500 ppm
I20260923 17:38:05.928812 4175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:05 dist-test-slave-ppjl krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185085, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:05.932075 4175 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:38:05.933529 4175 webserver.cc:460] Webserver started at http://127.20.240.66:36207/ using document root <none> and password file <none>
I20260923 17:38:05.934219 4175 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:05.934324 4175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:05.940039 4175 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:38:05.943145 4190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:05.944784 4175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:38:05.944953 4175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "a255d4034a1247838478b91846b78652"
format_stamp: "Formatted at 2026-09-23 17:38:00 on dist-test-slave-ppjl"
server_key: "0e22905ad33450bdcdde0bf612ca2a2d"
server_key_iv: "e4d8ca6a13c3ea27ade1aeee27650b78"
server_key_version: "encryptionkey@0"
I20260923 17:38:05.945986 4175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:05.965144 4175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:05.969575 4175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:05.969828 4175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:05.970827 4175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:38:05.972178 4175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:38:05.972272 4175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:05.972399 4175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:38:05.972463 4175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:06.005081 4175 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:44359
I20260923 17:38:06.005137 4302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:44359 every 8 connection(s)
I20260923 17:38:06.006527 4175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:38:06.011636 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:06.009635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:35433)
0923 17:38:06.010457 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:06.010678 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":352,"threads_started":1}
W20260923 17:38:06.012347 4303 heartbeater.cc:644] Failed to heartbeat to 127.20.240.126:41471 (0 consecutive failures): Network error: Failed to ping master at 127.20.240.126:41471: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
I20260923 17:38:06.013224 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:06.012884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:47777)
0923 17:38:06.013048 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:06.013118 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 17:38:06.013523 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 4175
I20260923 17:38:06.014101 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:06.013770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:60563)
0923 17:38:06.013923 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:06.013996 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 17:38:06.014117 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:44211
--local_ip_for_outbound_sockets=127.20.240.67
--tserver_master_addrs=127.20.240.126:41471
--webserver_port=38179
--webserver_interface=127.20.240.67
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:38:06.014379 4303 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:38:06.339546 4306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:06.339862 4306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:06.339988 4306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:06.347281 4306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:06.347404 4306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:06.347558 4306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:38:06.357580 4306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:44211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=38179
--tserver_master_addrs=127.20.240.126:41471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:06.358932 4306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:06.360442 4306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:06.374168 4314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:06.374233 4312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:06.374207 4311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:06.375115 4306 server_base.cc:1034] running on GCE node
I20260923 17:38:06.375779 4306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:06.376888 4306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:06.378108 4306 hybrid_clock.cc:648] HybridClock initialized: now 1790185086378036 us; error 51 us; skew 500 ppm
I20260923 17:38:06.378458 4306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:06 dist-test-slave-ppjl krb5kdc[2858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185086, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:06.381821 4306 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:38:06.383529 4306 webserver.cc:460] Webserver started at http://127.20.240.67:38179/ using document root <none> and password file <none>
I20260923 17:38:06.384290 4306 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:06.384431 4306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:06.390535 4306 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:38:06.393473 4321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:06.395208 4306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:38:06.395394 4306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "93a1ff7e14d64d418e9b4751ac279532"
format_stamp: "Formatted at 2026-09-23 17:38:00 on dist-test-slave-ppjl"
server_key: "7ea654838722b2a081ed3059704990d4"
server_key_iv: "53a169c046b5eee3a7d3df0f56a229fb"
server_key_version: "encryptionkey@0"
I20260923 17:38:06.396420 4306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:06.412122 4306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:06.416448 4306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:06.416714 4306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:06.417716 4306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:38:06.419122 4306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:38:06.419247 4306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260923 17:38:06.419363 4306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:38:06.419450 4306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:06.450455 4306 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:44211
I20260923 17:38:06.450511 4433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:44211 every 8 connection(s)
I20260923 17:38:06.452003 4306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:38:06.456470 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 4306
I20260923 17:38:06.457254 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:06.454995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:41601)
0923 17:38:06.455783 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:06.456032 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":321,"threads_started":1}
W20260923 17:38:06.457886 4434 heartbeater.cc:644] Failed to heartbeat to 127.20.240.126:41471 (0 consecutive failures): Network error: Failed to ping master at 127.20.240.126:41471: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
I20260923 17:38:06.458810 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:06.458463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:41237)
0923 17:38:06.458625 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:06.458698 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 17:38:06.459623 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:06.459313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:41317)
0923 17:38:06.459450 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:06.459522 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 17:38:06.459837 4434 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:38:06.566455 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:06.565593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:59991)
0923 17:38:06.566088 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:06.566197 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":260,"threads_started":1}
I20260923 17:38:07.015978 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:07.015013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:35873)
0923 17:38:07.015532 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:07.015664 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":267,"threads_started":1}
I20260923 17:38:07.461277 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:07.460439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:34763)
0923 17:38:07.460919 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:07.461049 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":225,"threads_started":1}
I20260923 17:38:07.568387 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:07.567563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:37797)
0923 17:38:07.568049 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:07.568181 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":249,"threads_started":1}
I20260923 17:38:08.017758 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:08.016995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:41229)
0923 17:38:08.017480 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:08.017573 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":243,"threads_started":1}
I20260923 17:38:08.462939 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:08.462227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:37835)
0923 17:38:08.462674 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:08.462760 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":227,"threads_started":1}
I20260923 17:38:08.570344 4444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:08.569469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:54607)
0923 17:38:08.570031 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:08.570132 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":252,"threads_started":1}
I20260923 17:38:09.019984 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:09.018885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:57457)
0923 17:38:09.019628 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:09.019742 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":444,"threads_started":1}
I20260923 17:38:09.464600 4446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:09.463837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:38569)
0923 17:38:09.464339 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:09.464428 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":274,"threads_started":1}
I20260923 17:38:09.572340 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:09.571406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:35663)
0923 17:38:09.572061 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:09.572162 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":398,"threads_started":1}
I20260923 17:38:10.021835 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:10.021098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:41073)
0923 17:38:10.021552 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:10.021631 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":217,"threads_started":1}
I20260923 17:38:10.466480 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:10.465634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:51663)
0923 17:38:10.466090 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:10.466196 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":223,"threads_started":1}
I20260923 17:38:10.574270 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:10.573445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:55109)
0923 17:38:10.573995 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:10.574089 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":269,"threads_started":1}
I20260923 17:38:11.023775 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:11.022906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:48735)
0923 17:38:11.023489 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:11.023587 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":325,"threads_started":1}
I20260923 17:38:11.468313 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:11.467541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:50483)
0923 17:38:11.468015 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:11.468103 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":237,"threads_started":1}
I20260923 17:38:11.576296 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:11.575357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:48937)
0923 17:38:11.575952 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:11.576053 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":325,"threads_started":1}
I20260923 17:38:12.025499 4454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:12.024730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:34559)
0923 17:38:12.025221 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:12.025325 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":256,"threads_started":1}
I20260923 17:38:12.470160 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:12.469357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:49057)
0923 17:38:12.469871 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:12.469984 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":281,"threads_started":1}
I20260923 17:38:12.578256 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:12.577385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:47163)
0923 17:38:12.577950 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:12.578035 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":270,"threads_started":1}
I20260923 17:38:13.027402 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:13.026518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:58279)
0923 17:38:13.027040 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:13.027129 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":232,"threads_started":1}
I20260923 17:38:13.472023 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:13.471153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:51475)
0923 17:38:13.471710 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:13.471812 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":221,"threads_started":1}
I20260923 17:38:13.580119 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:13.579307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:57415)
0923 17:38:13.579829 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:13.579920 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":249,"threads_started":1}
I20260923 17:38:14.029289 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:14.028428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:36325)
0923 17:38:14.029008 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:14.029104 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":322,"threads_started":1}
I20260923 17:38:14.473730 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:14.473001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:37331)
0923 17:38:14.473467 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:14.473555 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":256,"threads_started":1}
I20260923 17:38:14.581883 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:14.581110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:46795)
0923 17:38:14.581603 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:14.581703 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":228,"threads_started":1}
I20260923 17:38:15.031095 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:15.030291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:46173)
0923 17:38:15.030816 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:15.030914 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":242,"threads_started":1}
I20260923 17:38:15.475589 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:15.474749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:39877)
0923 17:38:15.475303 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:15.475391 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":257,"threads_started":1}
I20260923 17:38:15.583609 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:15.582862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:60325)
0923 17:38:15.583358 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:15.583440 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":239,"threads_started":1}
I20260923 17:38:16.032850 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:16.032067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:60827)
0923 17:38:16.032533 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:16.032619 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":243,"threads_started":1}
I20260923 17:38:16.477316 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:16.476597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:57709)
0923 17:38:16.477040 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:16.477127 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":230,"threads_started":1}
I20260923 17:38:16.585506 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:16.584661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:42167)
0923 17:38:16.585182 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:16.585304 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":306,"threads_started":1}
I20260923 17:38:17.034721 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:17.033926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:48891)
0923 17:38:17.034421 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:17.034528 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":239,"threads_started":1}
I20260923 17:38:17.479128 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:17.478294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:39507)
0923 17:38:17.478828 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:17.478943 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":264,"threads_started":1}
I20260923 17:38:17.587323 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:17.586476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:43487)
0923 17:38:17.587018 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:17.587110 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":324,"threads_started":1}
I20260923 17:38:18.036706 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:18.035939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:38253)
0923 17:38:18.036433 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:18.036529 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":249,"threads_started":1}
I20260923 17:38:18.481082 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:18.480233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:47501)
0923 17:38:18.480825 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:18.480916 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":287,"threads_started":1}
I20260923 17:38:18.589221 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:18.588341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:50221)
0923 17:38:18.588869 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:18.588982 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":275,"threads_started":1}
I20260923 17:38:19.039029 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:19.037684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:41097)
0923 17:38:19.038644 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:19.038773 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":804,"thread_start_us":422,"threads_started":1}
I20260923 17:38:19.482901 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:19.482101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:50003)
0923 17:38:19.482553 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:19.482667 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":221,"threads_started":1}
I20260923 17:38:19.590845 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:19.590102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:49955)
0923 17:38:19.590519 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:19.590606 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":205,"threads_started":1}
I20260923 17:38:20.041133 4478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:20.040229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:33075)
0923 17:38:20.040767 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:20.040898 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":255,"threads_started":1}
I20260923 17:38:20.484669 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:20.483860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:37167)
0923 17:38:20.484379 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:20.484467 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":281,"threads_started":1}
I20260923 17:38:20.592748 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:20.591965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.65:43541)
0923 17:38:20.592410 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:20.592509 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":217,"threads_started":1}
I20260923 17:38:21.043192 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:21.042346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.66:44641)
0923 17:38:21.042856 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:21.042965 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":256,"threads_started":1}
I20260923 17:38:21.468726 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 4008
I20260923 17:38:21.486690 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:21.485805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41471 (local address 127.20.240.67:53329)
0923 17:38:21.486368 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:21.486484 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:41471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":301,"threads_started":1}
I20260923 17:38:21.486968 4482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:21.469857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.20.240.65:46557)
0923 17:38:21.470747 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:21.470766 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:21.470821 (+ 55us) server_negotiation.cc:407] Connection header received
0923 17:38:21.472786 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:21.472796 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:21.473224 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:21.473523 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:21.474935 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:21.476215 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:21.477475 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:21.478141 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:21.478275 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:21.481036 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:21.481043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:21.481059 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:21.481098 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:21.483860 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:21.484512 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:21.484518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:21.484524 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:21.484682 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:21.485055 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:21.485058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:21.485060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:21.485609 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:21.485942 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:21.486231 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 17:38:21.486627 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":567,"threads_started":1}
I20260923 17:38:21.488350 4167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:21.572938 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 4008
I20260923 17:38:21.587211 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 4175
I20260923 17:38:21.606045 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:21.588248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.20.240.66:44359)
0923 17:38:21.588881 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:21.588895 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:21.588935 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:38:21.590640 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:21.590654 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:21.591111 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:21.591614 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:21.593120 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:21.594494 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:21.595968 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:21.596842 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:21.597013 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:21.599931 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:21.599940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:21.599957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:21.600000 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:21.602869 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:21.603583 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:21.603590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:21.603597 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:21.603820 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:21.604182 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:21.604186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:21.604188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:21.604773 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:21.605006 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:21.605375 (+ 369us) server_negotiation.cc:299] Negotiation successful
0923 17:38:21.605754 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":330,"threads_started":1}
I20260923 17:38:21.607479 4298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:21.691557 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 4175
I20260923 17:38:21.706560 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 4306
I20260923 17:38:21.725841 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:21.707706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33856 (local address 127.20.240.67:44211)
0923 17:38:21.708368 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:21.708383 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:21.708426 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:38:21.710077 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:21.710089 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:21.710537 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:21.710886 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:21.712339 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:21.713665 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:21.715061 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:21.715876 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:21.716018 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:21.719147 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:21.719191 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:21.719214 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:21.719282 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:21.722274 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:21.723182 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:21.723193 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:21.723203 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:21.723383 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:21.723840 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:21.723844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:21.723846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:21.724376 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:21.724671 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:21.725071 (+ 400us) server_negotiation.cc:299] Negotiation successful
0923 17:38:21.725486 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":326,"threads_started":1}
I20260923 17:38:21.727231 4429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:21.811954 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 4306
2026-09-23T17:38:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24420 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:38:21 dist-test-slave-ppjl krb5kdc[4491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:38:21 dist-test-slave-ppjl krb5kdc[4491](info): set up 2 sockets
Sep 23 17:38:21 dist-test-slave-ppjl krb5kdc[4491](info): commencing operation
krb5kdc: starting...
W20260923 17:38:22.832355 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:38:22 dist-test-slave-ppjl krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:38:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:38:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:38:23.072273 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:35957
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:35957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:38:23.406044 4504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:23.406345 4504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:23.406461 4504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:23.413862 4504 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 17:38:23.413967 4504 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 17:38:23.414005 4504 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 17:38:23.414073 4504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:38:23.414142 4504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:23.414237 4504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:23.414292 4504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:38:23.414367 4504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:38:23.424635 4504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:35957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:35957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:23.426014 4504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:23.427399 4504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:23.439333 4509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:23.439394 4512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:23.439374 4510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:23.440385 4504 server_base.cc:1034] running on GCE node
I20260923 17:38:23.441182 4504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:23.442667 4504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:23.443895 4504 hybrid_clock.cc:648] HybridClock initialized: now 1790185103443813 us; error 51 us; skew 500 ppm
I20260923 17:38:23.444269 4504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:23 dist-test-slave-ppjl krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185103, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:23.447548 4504 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:38:23.449112 4504 webserver.cc:460] Webserver started at http://127.20.240.126:37831/ using document root <none> and password file <none>
I20260923 17:38:23.449816 4504 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:23.449918 4504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:23.450233 4504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:38:23.453318 4504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "cd405a1902e64b24bbb1e81cc8ffbc43"
format_stamp: "Formatted at 2026-09-23 17:38:23 on dist-test-slave-ppjl"
server_key: "5be8f024338aca1b6c511f90e93e7fb4"
server_key_iv: "8f788692fb1fcb0f26ebf482200d5b3e"
server_key_version: "encryptionkey@0"
I20260923 17:38:23.454098 4504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "cd405a1902e64b24bbb1e81cc8ffbc43"
format_stamp: "Formatted at 2026-09-23 17:38:23 on dist-test-slave-ppjl"
server_key: "5be8f024338aca1b6c511f90e93e7fb4"
server_key_iv: "8f788692fb1fcb0f26ebf482200d5b3e"
server_key_version: "encryptionkey@0"
I20260923 17:38:23.459589 4504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260923 17:38:23.463304 4519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:23.464932 4504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:38:23.465116 4504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "cd405a1902e64b24bbb1e81cc8ffbc43"
format_stamp: "Formatted at 2026-09-23 17:38:23 on dist-test-slave-ppjl"
server_key: "5be8f024338aca1b6c511f90e93e7fb4"
server_key_iv: "8f788692fb1fcb0f26ebf482200d5b3e"
server_key_version: "encryptionkey@0"
I20260923 17:38:23.465405 4504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:23.480201 4504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:23.484355 4504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:23.484612 4504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:23.503130 4504 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:35957
I20260923 17:38:23.503149 4570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:35957 every 8 connection(s)
I20260923 17:38:23.504655 4504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:38:23.505843 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 4504
I20260923 17:38:23.506090 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:38:23.506924 21441 external_mini_cluster.cc:1428] Setting key 71c2da0e19a0e031467b35bac314559e
I20260923 17:38:23.509398 4571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:38:23.520594 4571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Bootstrap starting.
Sep 23 17:38:23 dist-test-slave-ppjl krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:38:23.524102 4571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Neither blocks nor log segments found. Creating new log.
I20260923 17:38:23.525204 4571 log.cc:824] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:23.528081 4571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: No bootstrap required, opened a new log
I20260923 17:38:23.529309 4574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:23.509064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.20.240.126:35957)
0923 17:38:23.509880 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:23.509892 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:23.509932 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:38:23.511639 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:23.511666 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:23.512070 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:23.512450 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:23.513921 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:23.515308 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:23.516608 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:23.517323 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:23.517507 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:23.521862 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:23.521871 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:23.521886 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:23.521933 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:23.525189 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:23.525916 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:23.525923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:23.525929 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:23.526116 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:23.526502 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:23.526506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:23.526508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:23.527032 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:23.527324 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:23.527822 (+ 498us) server_negotiation.cc:299] Negotiation successful
0923 17:38:23.528288 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":361,"threads_started":1}
I20260923 17:38:23.536500 4571 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } }
I20260923 17:38:23.536886 4571 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:38:23.536967 4571 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd405a1902e64b24bbb1e81cc8ffbc43, State: Initialized, Role: FOLLOWER
I20260923 17:38:23.537516 4571 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } }
I20260923 17:38:23.537729 4571 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:38:23.537827 4571 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:38:23.537961 4571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:38:23.540162 4571 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } }
I20260923 17:38:23.540619 4571 leader_election.cc:302] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd405a1902e64b24bbb1e81cc8ffbc43; no voters:
I20260923 17:38:23.541217 4571 leader_election.cc:288] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:38:23.541363 4576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:38:23.542475 4576 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 1 LEADER]: Becoming Leader. State: Replica: cd405a1902e64b24bbb1e81cc8ffbc43, State: Running, Role: LEADER
I20260923 17:38:23.543193 4576 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } }
I20260923 17:38:23.543414 4571 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:38:23.546228 4577 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } } }
I20260923 17:38:23.546721 4577 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [sys.catalog]: This master's current role is: LEADER
I20260923 17:38:23.546274 4578 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd405a1902e64b24bbb1e81cc8ffbc43. Latest consensus state: current_term: 1 leader_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } } }
I20260923 17:38:23.546948 4578 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [sys.catalog]: This master's current role is: LEADER
I20260923 17:38:23.547686 4585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:38:23.552946 4585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:38:23.565706 4585 catalog_manager.cc:1347] Generated new cluster ID: bd956bd3b52b4e00a09799c21c3c9b76
I20260923 17:38:23.565825 4585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:38:23.574330 4585 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:38:23.576333 4585 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:38:23.586362 4585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Generated new TSK 0
I20260923 17:38:23.587319 4585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:38:23.663251 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:35957
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:38:24.000106 4597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:24.000397 4597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:24.000519 4597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:24.007891 4597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:24.008005 4597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:24.008160 4597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:38:24.018139 4597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:24.019551 4597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:24.020925 4597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:24.035208 4602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:24.035321 4605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:24.035215 4603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:24.036459 4597 server_base.cc:1034] running on GCE node
I20260923 17:38:24.037216 4597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:24.038375 4597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:24.039645 4597 hybrid_clock.cc:648] HybridClock initialized: now 1790185104039575 us; error 75 us; skew 500 ppm
I20260923 17:38:24.040083 4597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:24 dist-test-slave-ppjl krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185104, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:24.046939 4597 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:38:24.048731 4597 webserver.cc:460] Webserver started at http://127.20.240.65:37879/ using document root <none> and password file <none>
I20260923 17:38:24.049535 4597 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:24.049655 4597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:24.049969 4597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:38:24.053153 4597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "c19171edd1ba4b3d8dc6bca6775ea599"
format_stamp: "Formatted at 2026-09-23 17:38:24 on dist-test-slave-ppjl"
server_key: "dd5084ab16373d8c29dc9fb9490e41a0"
server_key_iv: "f15942c4199dd9882b016a99d0ec3699"
server_key_version: "encryptionkey@0"
I20260923 17:38:24.053938 4597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "c19171edd1ba4b3d8dc6bca6775ea599"
format_stamp: "Formatted at 2026-09-23 17:38:24 on dist-test-slave-ppjl"
server_key: "dd5084ab16373d8c29dc9fb9490e41a0"
server_key_iv: "f15942c4199dd9882b016a99d0ec3699"
server_key_version: "encryptionkey@0"
I20260923 17:38:24.059856 4597 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260923 17:38:24.063889 4612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:24.065618 4597 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:38:24.065809 4597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "c19171edd1ba4b3d8dc6bca6775ea599"
format_stamp: "Formatted at 2026-09-23 17:38:24 on dist-test-slave-ppjl"
server_key: "dd5084ab16373d8c29dc9fb9490e41a0"
server_key_iv: "f15942c4199dd9882b016a99d0ec3699"
server_key_version: "encryptionkey@0"
I20260923 17:38:24.066020 4597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:24.083644 4597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:24.088168 4597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:24.088451 4597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:24.089417 4597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:38:24.090783 4597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:38:24.090863 4597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:24.090994 4597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:38:24.091055 4597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:24.121757 4597 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:34955
I20260923 17:38:24.121797 4724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:34955 every 8 connection(s)
I20260923 17:38:24.123070 4597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:38:24.130885 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 4597
I20260923 17:38:24.131111 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:38:24.131803 21441 external_mini_cluster.cc:1428] Setting key f77aae813c1d17a603f6b59363246b8a
Sep 23 17:38:24 dist-test-slave-ppjl krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185104, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:38:24.144878 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:24.125803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:52869 (local address 127.20.240.126:35957)
0923 17:38:24.126268 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:24.126272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:24.127181 (+ 909us) server_negotiation.cc:407] Connection header received
0923 17:38:24.129019 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:24.129025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:24.129121 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:24.129218 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:24.130906 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:24.131751 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:24.133453 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:24.133862 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:24.133988 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:24.137952 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:24.137959 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:24.137963 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:24.138004 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:24.141281 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:24.142399 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:24.142405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:24.142408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:24.142583 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:24.143147 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:24.143173 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:24.143177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:24.143470 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:24.143698 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:24.144336 (+ 638us) server_negotiation.cc:299] Negotiation successful
0923 17:38:24.144521 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":224,"threads_started":1}
I20260923 17:38:24.146037 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:24.126215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:52869)
0923 17:38:24.126976 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:24.127014 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:24.128734 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:24.129501 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:24.129512 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:24.130086 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:38:24.130716 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:24.130736 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:24.131918 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:24.131922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:24.133312 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:24.133325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:24.134101 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:24.134108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:24.134345 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:24.134984 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:38:24.135012 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:38:24.137643 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:38:24.141539 (+ 3896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:24.141550 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:24.141580 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:24.142256 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:24.142729 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:24.142734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:24.142738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:24.143042 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:24.143608 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:38:24.143615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:38:24.143989 (+ 374us) client_negotiation.cc:769] Sending connection context
0923 17:38:24.144231 (+ 242us) client_negotiation.cc:240] Negotiation successful
0923 17:38:24.144648 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":326,"threads_started":1}
I20260923 17:38:24.147804 4725 heartbeater.cc:347] Connected to a master server at 127.20.240.126:35957
I20260923 17:38:24.148144 4725 heartbeater.cc:464] Registering TS with master...
I20260923 17:38:24.148885 4725 heartbeater.cc:511] Master 127.20.240.126:35957 requested a full tablet report, sending...
I20260923 17:38:24.151134 4536 ts_manager.cc:194] Registered new tserver with Master: c19171edd1ba4b3d8dc6bca6775ea599 (127.20.240.65:34955)
I20260923 17:38:24.153126 4536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:52869
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:38:24.190496 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:35957
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:38:24.511780 4731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:24.512030 4731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:24.512148 4731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:24.519491 4731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:24.519598 4731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:24.519744 4731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:38:24.529699 4731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:24.531008 4731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:24.532438 4731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:24.545750 4736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:24.545704 4737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:24.545670 4739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:24.551986 4731 server_base.cc:1034] running on GCE node
I20260923 17:38:24.552723 4731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:24.553753 4731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:24.554973 4731 hybrid_clock.cc:648] HybridClock initialized: now 1790185104554910 us; error 53 us; skew 500 ppm
I20260923 17:38:24.555335 4731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:24 dist-test-slave-ppjl krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185104, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:24.558445 4731 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:38:24.560168 4731 webserver.cc:460] Webserver started at http://127.20.240.66:32887/ using document root <none> and password file <none>
I20260923 17:38:24.560880 4731 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:24.561000 4731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:24.561290 4731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:38:24.564437 4731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "8b2354c55c5544a8bf4471195484345e"
format_stamp: "Formatted at 2026-09-23 17:38:24 on dist-test-slave-ppjl"
server_key: "041540b8cb8ca7e50b8ed085da785dc1"
server_key_iv: "e576e3bdb5de60796cb898e0e7144070"
server_key_version: "encryptionkey@0"
I20260923 17:38:24.565204 4731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "8b2354c55c5544a8bf4471195484345e"
format_stamp: "Formatted at 2026-09-23 17:38:24 on dist-test-slave-ppjl"
server_key: "041540b8cb8ca7e50b8ed085da785dc1"
server_key_iv: "e576e3bdb5de60796cb898e0e7144070"
server_key_version: "encryptionkey@0"
I20260923 17:38:24.570798 4731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:38:24.574590 4746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:24.576107 4731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:38:24.576270 4731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "8b2354c55c5544a8bf4471195484345e"
format_stamp: "Formatted at 2026-09-23 17:38:24 on dist-test-slave-ppjl"
server_key: "041540b8cb8ca7e50b8ed085da785dc1"
server_key_iv: "e576e3bdb5de60796cb898e0e7144070"
server_key_version: "encryptionkey@0"
I20260923 17:38:24.576479 4731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:24.608459 4731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:24.612936 4731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:24.613220 4731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:24.614218 4731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:38:24.615751 4731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:38:24.615873 4731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:24.616036 4731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:38:24.616110 4731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:24.650096 4731 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:42977
I20260923 17:38:24.650115 4858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:42977 every 8 connection(s)
I20260923 17:38:24.651626 4731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:38:24.653192 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 4731
I20260923 17:38:24.653375 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:38:24.653867 21441 external_mini_cluster.cc:1428] Setting key 2e3f6a92e1a68dcf21a4faaff05277eb
Sep 23 17:38:24 dist-test-slave-ppjl krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185104, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:38:24.674046 4861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:24.654306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:54507 (local address 127.20.240.126:35957)
0923 17:38:24.655080 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:24.655086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:24.658413 (+ 3327us) server_negotiation.cc:407] Connection header received
0923 17:38:24.658484 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:24.658500 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:24.658581 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:24.658681 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:24.660444 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:24.661513 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:24.663211 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:24.663826 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:24.663951 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:24.667819 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:24.667825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:24.667828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:24.667865 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:24.670573 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:24.671640 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:24.671645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:24.671647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:24.671796 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:24.672264 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:24.672266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:24.672268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:24.672541 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:24.672664 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:24.673573 (+ 909us) server_negotiation.cc:299] Negotiation successful
0923 17:38:24.673786 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":524,"threads_started":1}
I20260923 17:38:24.675699 4862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:24.654848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:54507)
0923 17:38:24.655666 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:24.655719 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:24.658020 (+ 2301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:24.658955 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:24.658965 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:24.659635 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:38:24.660242 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:24.660262 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:24.661690 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:24.661694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:24.663042 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:24.663056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:24.664082 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:24.664090 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:24.664308 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:24.665060 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:38:24.665085 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:38:24.667544 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:38:24.670748 (+ 3204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:24.670759 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:24.670779 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:24.671471 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:24.671898 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:24.671902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:24.671905 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:24.672163 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:24.672670 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:38:24.672679 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:38:24.673150 (+ 471us) client_negotiation.cc:769] Sending connection context
0923 17:38:24.673458 (+ 308us) client_negotiation.cc:240] Negotiation successful
0923 17:38:24.674044 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":347,"threads_started":1}
I20260923 17:38:24.677241 4859 heartbeater.cc:347] Connected to a master server at 127.20.240.126:35957
I20260923 17:38:24.677551 4859 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
I20260923 17:38:24.678337 4859 heartbeater.cc:511] Master 127.20.240.126:35957 requested a full tablet report, sending...
I20260923 17:38:24.679983 4536 ts_manager.cc:194] Registered new tserver with Master: 8b2354c55c5544a8bf4471195484345e (127.20.240.66:42977)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
I20260923 17:38:24.681006 4536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:54507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:38:24.708319 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:35957
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:38:25.042621 4865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:25.042951 4865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:25.043128 4865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:25.051512 4865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:25.051644 4865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:25.051818 4865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:38:25.062036 4865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:25.063500 4865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:25.064878 4865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:25.079005 4871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:25.079329 4870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:25.079687 4873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:25.079949 4865 server_base.cc:1034] running on GCE node
I20260923 17:38:25.080686 4865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:25.081759 4865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:25.082986 4865 hybrid_clock.cc:648] HybridClock initialized: now 1790185105082917 us; error 57 us; skew 500 ppm
I20260923 17:38:25.083372 4865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:25 dist-test-slave-ppjl krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185105, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:25.086577 4865 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:38:25.088208 4865 webserver.cc:460] Webserver started at http://127.20.240.67:43989/ using document root <none> and password file <none>
I20260923 17:38:25.088891 4865 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:25.089001 4865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:25.089301 4865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:38:25.092366 4865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "0461c3dc460541de93c36d9e0f788184"
format_stamp: "Formatted at 2026-09-23 17:38:25 on dist-test-slave-ppjl"
server_key: "4a5000bebe02edd07c57c04c7f526b81"
server_key_iv: "49b209950ed654a66ea9a2bc79e3f023"
server_key_version: "encryptionkey@0"
I20260923 17:38:25.093183 4865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "0461c3dc460541de93c36d9e0f788184"
format_stamp: "Formatted at 2026-09-23 17:38:25 on dist-test-slave-ppjl"
server_key: "4a5000bebe02edd07c57c04c7f526b81"
server_key_iv: "49b209950ed654a66ea9a2bc79e3f023"
server_key_version: "encryptionkey@0"
I20260923 17:38:25.098874 4865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:38:25.103262 4880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:25.105278 4865 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:38:25.105513 4865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "0461c3dc460541de93c36d9e0f788184"
format_stamp: "Formatted at 2026-09-23 17:38:25 on dist-test-slave-ppjl"
server_key: "4a5000bebe02edd07c57c04c7f526b81"
server_key_iv: "49b209950ed654a66ea9a2bc79e3f023"
server_key_version: "encryptionkey@0"
I20260923 17:38:25.105755 4865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:25.130911 4865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:25.135506 4865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:25.135784 4865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:25.136891 4865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:38:25.138296 4865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:38:25.138381 4865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:25.138497 4865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:38:25.138552 4865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:25.156913 4725 heartbeater.cc:503] Master 127.20.240.126:35957 was elected leader, sending a full tablet report...
I20260923 17:38:25.180500 4865 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:37297
I20260923 17:38:25.180583 4993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:37297 every 8 connection(s)
I20260923 17:38:25.181883 4865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:38:25.183231 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 4865
I20260923 17:38:25.183413 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:38:25.183867 21441 external_mini_cluster.cc:1428] Setting key 607a2a949428c7fa567dea66557841ab
Sep 23 17:38:25 dist-test-slave-ppjl krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185105, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:38:25.206655 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.186014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:46707 (local address 127.20.240.126:35957)
0923 17:38:25.191546 (+ 5532us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:25.191552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:25.191569 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:38:25.191622 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:25.191625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:25.191688 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:25.191788 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:25.193575 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.194393 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.196000 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.196418 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.196526 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.200358 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:25.200362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:25.200366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:25.200397 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:25.203012 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:25.203969 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:25.203973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:25.203976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:25.204106 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:25.204621 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:25.204625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:25.204627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:25.204913 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:25.205051 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:25.206023 (+ 972us) server_negotiation.cc:299] Negotiation successful
0923 17:38:25.206264 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5416,"thread_start_us":5280,"threads_started":1}
I20260923 17:38:25.208031 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.185434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:46707)
0923 17:38:25.186254 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:25.186298 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:25.188245 (+ 1947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:25.192112 (+ 3867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:25.192122 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:25.192776 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:38:25.193387 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:25.193406 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.194556 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.194560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:25.195861 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:25.195874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.196649 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.196656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:25.196879 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.197497 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:38:25.197520 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:38:25.200150 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:38:25.203177 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:25.203188 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:25.203209 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:25.203835 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:25.204205 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:25.204209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:25.204212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:25.204466 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:25.205048 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:38:25.205057 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:38:25.205532 (+ 475us) client_negotiation.cc:769] Sending connection context
0923 17:38:25.205842 (+ 310us) client_negotiation.cc:240] Negotiation successful
0923 17:38:25.206511 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":296,"threads_started":1}
I20260923 17:38:25.209609 4994 heartbeater.cc:347] Connected to a master server at 127.20.240.126:35957
I20260923 17:38:25.209929 4994 heartbeater.cc:464] Registering TS with master...
I20260923 17:38:25.210819 4994 heartbeater.cc:511] Master 127.20.240.126:35957 requested a full tablet report, sending...
I20260923 17:38:25.212522 4535 ts_manager.cc:194] Registered new tserver with Master: 0461c3dc460541de93c36d9e0f788184 (127.20.240.67:37297)
I20260923 17:38:25.213485 4535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:46707
I20260923 17:38:25.225847 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:38:25.243674 4997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.230523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.20.240.126:35957)
0923 17:38:25.230743 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:25.230751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:25.231140 (+ 389us) server_negotiation.cc:407] Connection header received
0923 17:38:25.231384 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:25.231388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:25.231458 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:25.231557 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:25.233115 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.233856 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.235238 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.235665 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.235782 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.238363 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:25.238369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:25.238372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:25.238408 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:25.241033 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:25.241733 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:25.241739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:25.241741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:25.241869 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:25.242228 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:25.242231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:25.242233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:25.242557 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:25.242783 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:25.243118 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 17:38:25.243375 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:38:25.247910 4535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:38:25.254876 4535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:38:25.278852 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.268072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40700 (local address 127.20.240.66:42977)
0923 17:38:25.270379 (+ 2307us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:25.270386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:25.270406 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:38:25.270489 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:25.270495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:25.270668 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:25.270867 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:25.271618 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.273094 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.275959 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.277315 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.278056 (+ 741us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.278272 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:25.278366 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 17:38:25.278527 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2174,"thread_start_us":2056,"threads_started":1}
I20260923 17:38:25.278887 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.267058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:42977 (local address 127.0.0.1:40700)
0923 17:38:25.268234 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:25.268286 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:25.268475 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:25.270870 (+ 2395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:25.270877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:25.270906 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:25.271446 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:25.271458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.273239 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.273244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:25.275807 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:25.275821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.277584 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.277591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:25.277973 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.278127 (+ 154us) client_negotiation.cc:769] Sending connection context
0923 17:38:25.278197 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 17:38:25.278428 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":119,"thread_start_us":742,"threads_started":1}
I20260923 17:38:25.278998 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.269132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.20.240.65:34955)
0923 17:38:25.270603 (+ 1471us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:25.270609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:25.270631 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:38:25.270734 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:25.270742 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:25.270969 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:25.271192 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:25.271997 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.273583 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.275667 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.277027 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.277180 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.277417 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:25.278477 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0923 17:38:25.278635 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1319,"thread_start_us":1194,"threads_started":1}
I20260923 17:38:25.279002 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.267698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44770 (local address 127.20.240.67:37297)
0923 17:38:25.270291 (+ 2593us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:25.270300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:25.270331 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:38:25.270423 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:25.270430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:25.270684 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:25.270863 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:25.272015 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.273650 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.276149 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.277459 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.277587 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.277824 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:25.278355 (+ 531us) server_negotiation.cc:299] Negotiation successful
0923 17:38:25.278630 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2424,"thread_start_us":2279,"threads_started":1}
I20260923 17:38:25.279663 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.267207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:37297 (local address 127.0.0.1:44770)
0923 17:38:25.269126 (+ 1919us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:25.269150 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:25.269254 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:25.271104 (+ 1850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:25.271109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:25.271144 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:25.271779 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:25.271792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.273751 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.273756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:25.276019 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:25.276029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.277677 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.277684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:25.278068 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.278111 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 17:38:25.278207 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 17:38:25.279264 (+ 1057us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1808,"mutex_wait_us":917,"spinlock_wait_cycles":2304,"thread_start_us":1559,"threads_started":1}
I20260923 17:38:25.280208 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.268578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34955 (local address 127.0.0.1:56500)
0923 17:38:25.269762 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:25.269778 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:25.269881 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:25.271185 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:25.271189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:25.271210 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:25.271792 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:25.271807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.273770 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.273775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:25.275519 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:25.275531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.277553 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.277563 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:25.277904 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.278047 (+ 143us) client_negotiation.cc:769] Sending connection context
0923 17:38:25.278183 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 17:38:25.279917 (+ 1734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"mutex_wait_us":1662,"spinlock_wait_cycles":2432,"thread_start_us":890,"threads_started":1}
I20260923 17:38:25.283214 4794 tablet_service.cc:1467] Processing CreateTablet for tablet 37c4255d803f4f57a7a9c711e6fab376 (DEFAULT_TABLE table=test-table [id=38a79805aea2469db8ac4fe8f714312b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:38:25.283756 4929 tablet_service.cc:1467] Processing CreateTablet for tablet 37c4255d803f4f57a7a9c711e6fab376 (DEFAULT_TABLE table=test-table [id=38a79805aea2469db8ac4fe8f714312b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:38:25.284090 4660 tablet_service.cc:1467] Processing CreateTablet for tablet 37c4255d803f4f57a7a9c711e6fab376 (DEFAULT_TABLE table=test-table [id=38a79805aea2469db8ac4fe8f714312b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:38:25.284606 4794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c4255d803f4f57a7a9c711e6fab376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:38:25.285285 4929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c4255d803f4f57a7a9c711e6fab376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:38:25.285465 4660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c4255d803f4f57a7a9c711e6fab376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:38:25.296458 5013 tablet_bootstrap.cc:492] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e: Bootstrap starting.
I20260923 17:38:25.299806 5013 tablet_bootstrap.cc:654] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e: Neither blocks nor log segments found. Creating new log.
I20260923 17:38:25.300474 5014 tablet_bootstrap.cc:492] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184: Bootstrap starting.
I20260923 17:38:25.301312 5013 log.cc:824] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:25.303841 5015 tablet_bootstrap.cc:492] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599: Bootstrap starting.
I20260923 17:38:25.304299 5014 tablet_bootstrap.cc:654] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184: Neither blocks nor log segments found. Creating new log.
I20260923 17:38:25.305796 5014 log.cc:824] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:25.306820 5015 tablet_bootstrap.cc:654] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599: Neither blocks nor log segments found. Creating new log.
I20260923 17:38:25.308120 5015 log.cc:824] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:25.308888 5014 tablet_bootstrap.cc:492] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184: No bootstrap required, opened a new log
I20260923 17:38:25.309078 5013 tablet_bootstrap.cc:492] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e: No bootstrap required, opened a new log
I20260923 17:38:25.309175 5014 ts_tablet_manager.cc:1397] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 17:38:25.309317 5013 ts_tablet_manager.cc:1397] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260923 17:38:25.311004 5015 tablet_bootstrap.cc:492] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599: No bootstrap required, opened a new log
I20260923 17:38:25.311374 5015 ts_tablet_manager.cc:1397] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260923 17:38:25.316680 5013 raft_consensus.cc:348] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } }
I20260923 17:38:25.317080 5013 raft_consensus.cc:374] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:38:25.317198 5013 raft_consensus.cc:729] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2354c55c5544a8bf4471195484345e, State: Initialized, Role: FOLLOWER
I20260923 17:38:25.317935 5013 consensus_queue.cc:260] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } }
I20260923 17:38:25.319046 4859 heartbeater.cc:503] Master 127.20.240.126:35957 was elected leader, sending a full tablet report...
I20260923 17:38:25.319845 5013 ts_tablet_manager.cc:1428] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260923 17:38:25.319805 5014 raft_consensus.cc:348] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } }
I20260923 17:38:25.320246 5014 raft_consensus.cc:374] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:38:25.320341 5014 raft_consensus.cc:729] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0461c3dc460541de93c36d9e0f788184, State: Initialized, Role: FOLLOWER
I20260923 17:38:25.321035 5014 consensus_queue.cc:260] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } }
I20260923 17:38:25.321743 5015 raft_consensus.cc:348] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } }
I20260923 17:38:25.322336 5015 raft_consensus.cc:374] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:38:25.322443 5015 raft_consensus.cc:729] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19171edd1ba4b3d8dc6bca6775ea599, State: Initialized, Role: FOLLOWER
I20260923 17:38:25.323098 5015 consensus_queue.cc:260] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } }
I20260923 17:38:25.324431 4994 heartbeater.cc:503] Master 127.20.240.126:35957 was elected leader, sending a full tablet report...
I20260923 17:38:25.324790 5015 ts_tablet_manager.cc:1428] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260923 17:38:25.325487 5014 ts_tablet_manager.cc:1428] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260923 17:38:25.375536 4726 tablet.cc:2369] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:38:25.403878 4860 tablet.cc:2369] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:38:25.435075 4995 tablet.cc:2369] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:38:25.514766 5019 raft_consensus.cc:482] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:38:25.515074 5019 raft_consensus.cc:504] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } }
I20260923 17:38:25.516896 5019 leader_election.cc:288] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0461c3dc460541de93c36d9e0f788184 (127.20.240.67:37297), c19171edd1ba4b3d8dc6bca6775ea599 (127.20.240.65:34955)
I20260923 17:38:25.523778 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.517325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:46403 (local address 127.20.240.67:37297)
0923 17:38:25.517533 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:25.517539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:25.518316 (+ 777us) server_negotiation.cc:407] Connection header received
0923 17:38:25.518540 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:25.518547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:25.518629 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:25.518711 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:25.519322 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.520216 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.521998 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.522786 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.522931 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.523010 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:25.523388 (+ 378us) server_negotiation.cc:299] Negotiation successful
0923 17:38:25.523529 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:38:25.523806 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.517398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:32775 (local address 127.20.240.65:34955)
0923 17:38:25.517582 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:25.517587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:25.518592 (+ 1005us) server_negotiation.cc:407] Connection header received
0923 17:38:25.518738 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:25.518744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:25.518819 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:25.518913 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:25.519538 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.520246 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.522025 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.522792 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.522941 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.523014 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:25.523423 (+ 409us) server_negotiation.cc:299] Negotiation successful
0923 17:38:25.523553 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:38:25.523957 5023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.517383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:37297 (local address 127.20.240.66:46403)
0923 17:38:25.518183 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:25.518206 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:25.518383 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:25.518743 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:25.518747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:25.518773 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:25.519144 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:25.519176 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.520345 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.520350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:25.521852 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:25.521864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.523025 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.523032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:25.523208 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.523269 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 17:38:25.523342 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 17:38:25.523472 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":19,"thread_start_us":745,"threads_started":1}
I20260923 17:38:25.524001 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.517383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34955 (local address 127.20.240.66:32775)
0923 17:38:25.518477 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:25.518493 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:25.518620 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:25.518938 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:25.518941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:25.518961 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:25.519375 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:25.519389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.520389 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.520396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:25.521874 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:25.521886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.523066 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.523071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:25.523272 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.523296 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:38:25.523381 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 17:38:25.523508 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":92,"spinlock_wait_cycles":19712,"thread_start_us":402,"threads_started":1}
I20260923 17:38:25.525058 4949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c4255d803f4f57a7a9c711e6fab376" candidate_uuid: "8b2354c55c5544a8bf4471195484345e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0461c3dc460541de93c36d9e0f788184" is_pre_election: true
I20260923 17:38:25.525063 4680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c4255d803f4f57a7a9c711e6fab376" candidate_uuid: "8b2354c55c5544a8bf4471195484345e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" is_pre_election: true
I20260923 17:38:25.525533 4949 raft_consensus.cc:2454] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b2354c55c5544a8bf4471195484345e in term 0.
I20260923 17:38:25.525537 4680 raft_consensus.cc:2454] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b2354c55c5544a8bf4471195484345e in term 0.
I20260923 17:38:25.526175 4748 leader_election.cc:302] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0461c3dc460541de93c36d9e0f788184, 8b2354c55c5544a8bf4471195484345e; no voters:
I20260923 17:38:25.526542 5019 raft_consensus.cc:2781] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:38:25.526677 5019 raft_consensus.cc:482] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:38:25.526772 5019 raft_consensus.cc:3037] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:38:25.529093 5019 raft_consensus.cc:504] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } }
I20260923 17:38:25.529729 5019 leader_election.cc:288] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [CANDIDATE]: Term 1 election: Requested vote from peers 0461c3dc460541de93c36d9e0f788184 (127.20.240.67:37297), c19171edd1ba4b3d8dc6bca6775ea599 (127.20.240.65:34955)
I20260923 17:38:25.530262 4680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c4255d803f4f57a7a9c711e6fab376" candidate_uuid: "8b2354c55c5544a8bf4471195484345e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19171edd1ba4b3d8dc6bca6775ea599"
I20260923 17:38:25.530313 4949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c4255d803f4f57a7a9c711e6fab376" candidate_uuid: "8b2354c55c5544a8bf4471195484345e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0461c3dc460541de93c36d9e0f788184"
I20260923 17:38:25.530467 4680 raft_consensus.cc:3037] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:38:25.530508 4949 raft_consensus.cc:3037] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:38:25.532711 4680 raft_consensus.cc:2454] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b2354c55c5544a8bf4471195484345e in term 1.
I20260923 17:38:25.532723 4949 raft_consensus.cc:2454] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b2354c55c5544a8bf4471195484345e in term 1.
I20260923 17:38:25.533156 4748 leader_election.cc:302] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0461c3dc460541de93c36d9e0f788184, 8b2354c55c5544a8bf4471195484345e; no voters:
I20260923 17:38:25.533445 5019 raft_consensus.cc:2781] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:38:25.533874 5019 raft_consensus.cc:686] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [term 1 LEADER]: Becoming Leader. State: Replica: 8b2354c55c5544a8bf4471195484345e, State: Running, Role: LEADER
I20260923 17:38:25.534339 5019 consensus_queue.cc:237] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } }
I20260923 17:38:25.538718 4535 catalog_manager.cc:5495] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b2354c55c5544a8bf4471195484345e (127.20.240.66). New cstate: current_term: 1 leader_uuid: "8b2354c55c5544a8bf4471195484345e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } health_report { overall_health: UNKNOWN } } }
I20260923 17:38:25.579989 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.574582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40702 (local address 127.20.240.66:42977)
0923 17:38:25.574840 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:25.574847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:25.574866 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:38:25.575060 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:25.575064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:25.575198 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:25.575332 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:38:25.575693 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.576415 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.577809 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.578234 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.578341 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.578838 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:38:25.579417 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:38:25.579549 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:25.579648 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 17:38:25.579732 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260923 17:38:25.588352 4949 raft_consensus.cc:1261] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184 [term 1 FOLLOWER]: Refusing update from remote peer 8b2354c55c5544a8bf4471195484345e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:38:25.588363 4680 raft_consensus.cc:1261] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599 [term 1 FOLLOWER]: Refusing update from remote peer 8b2354c55c5544a8bf4471195484345e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:38:25.589331 5019 consensus_queue.cc:1035] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:38:25.589692 5024 consensus_queue.cc:1035] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:38:25.617076 4535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:38:25.617350 4535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:38:25.621500 4535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Sending DeleteTablet for 3 replicas of tablet 37c4255d803f4f57a7a9c711e6fab376
I20260923 17:38:25.622884 4929 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c4255d803f4f57a7a9c711e6fab376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:38:25 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:44770
I20260923 17:38:25.623287 4794 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c4255d803f4f57a7a9c711e6fab376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:38:25 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:40700
I20260923 17:38:25.623274 4660 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c4255d803f4f57a7a9c711e6fab376 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:38:25 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:56500
I20260923 17:38:25.624235 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 4597
I20260923 17:38:25.638751 5037 tablet_replica.cc:330] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184: stopping tablet replica
I20260923 17:38:25.639895 5039 tablet_replica.cc:330] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599: stopping tablet replica
I20260923 17:38:25.640261 5037 raft_consensus.cc:2229] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:38:25.640347 5039 raft_consensus.cc:2229] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:38:25.640969 5037 raft_consensus.cc:2258] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:38:25.641000 5039 raft_consensus.cc:2258] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:38:25.641764 5038 tablet_replica.cc:330] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e: stopping tablet replica
I20260923 17:38:25.642199 5038 raft_consensus.cc:2229] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:38:25.642894 5038 raft_consensus.cc:2258] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:38:25.643503 5037 ts_tablet_manager.cc:1905] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:38:25.645364 5039 ts_tablet_manager.cc:1905] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:38:25.649250 5038 ts_tablet_manager.cc:1905] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:38:25.652309 5039 ts_tablet_manager.cc:1918] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:38:25.652593 5039 log.cc:1196] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/37c4255d803f4f57a7a9c711e6fab376
I20260923 17:38:25.653285 5039 ts_tablet_manager.cc:1939] T 37c4255d803f4f57a7a9c711e6fab376 P c19171edd1ba4b3d8dc6bca6775ea599: Deleting consensus metadata
I20260923 17:38:25.655633 4523 catalog_manager.cc:4841] TS c19171edd1ba4b3d8dc6bca6775ea599 (127.20.240.65:34955): tablet 37c4255d803f4f57a7a9c711e6fab376 (table test-table [id=38a79805aea2469db8ac4fe8f714312b]) successfully deleted
Sep 23 17:38:25 dist-test-slave-ppjl krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:38:25.658535 5038 ts_tablet_manager.cc:1918] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:38:25.658751 5038 log.cc:1196] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/37c4255d803f4f57a7a9c711e6fab376
I20260923 17:38:25.659348 5038 ts_tablet_manager.cc:1939] T 37c4255d803f4f57a7a9c711e6fab376 P 8b2354c55c5544a8bf4471195484345e: Deleting consensus metadata
I20260923 17:38:25.661070 4523 catalog_manager.cc:4841] TS 8b2354c55c5544a8bf4471195484345e (127.20.240.66:42977): tablet 37c4255d803f4f57a7a9c711e6fab376 (table test-table [id=38a79805aea2469db8ac4fe8f714312b]) successfully deleted
I20260923 17:38:25.662914 5037 ts_tablet_manager.cc:1918] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:38:25.663139 5037 log.cc:1196] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/37c4255d803f4f57a7a9c711e6fab376
I20260923 17:38:25.663697 5037 ts_tablet_manager.cc:1939] T 37c4255d803f4f57a7a9c711e6fab376 P 0461c3dc460541de93c36d9e0f788184: Deleting consensus metadata
I20260923 17:38:25.665503 4521 catalog_manager.cc:4841] TS 0461c3dc460541de93c36d9e0f788184 (127.20.240.67:37297): tablet 37c4255d803f4f57a7a9c711e6fab376 (table test-table [id=38a79805aea2469db8ac4fe8f714312b]) successfully deleted
I20260923 17:38:25.665681 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.625540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.20.240.65:34955)
0923 17:38:25.625738 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:25.625745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:25.639076 (+ 13331us) server_negotiation.cc:407] Connection header received
0923 17:38:25.639417 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:25.639427 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:25.639546 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:25.639671 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:25.642588 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.643798 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.645948 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.646612 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.646712 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.658215 (+ 11503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:25.658233 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:25.658252 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:25.658300 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:25.661949 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:25.662871 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:25.662882 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:25.662886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:25.663066 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:25.663619 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:25.663624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:25.663628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:25.664109 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:25.664388 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:25.665063 (+ 675us) server_negotiation.cc:299] Negotiation successful
0923 17:38:25.665302 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:38:25.667188 4720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:25.824615 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 4597
I20260923 17:38:25.844226 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 4731
Sep 23 17:38:25 dist-test-slave-ppjl krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:38:25.861171 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.844986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40704 (local address 127.20.240.66:42977)
0923 17:38:25.845164 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:25.845169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:25.845249 (+ 80us) server_negotiation.cc:407] Connection header received
0923 17:38:25.845436 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:25.845442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:25.845515 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:25.845611 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:25.847109 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.848066 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.849435 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.849880 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.849990 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:25.854750 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:25.854767 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:25.854783 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:25.854839 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:25.858235 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:25.858983 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:25.858991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:25.858997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:25.859176 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:25.859576 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:25.859580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:25.859582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:25.859961 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:25.860294 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:25.860594 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 17:38:25.860822 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:38:25.861915 4854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:25.977718 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 4731
I20260923 17:38:25.993057 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 4865
Sep 23 17:38:25 dist-test-slave-ppjl krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:38:26.008283 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:25.993856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.20.240.67:37297)
0923 17:38:25.994030 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:25.994037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:25.994057 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:38:25.994233 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:25.994237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:25.994302 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:25.994390 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:25.995669 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.996380 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.997607 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:25.997995 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:25.998085 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:26.002395 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:26.002412 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:26.002428 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:26.002464 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:26.005342 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:26.006055 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:26.006062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:26.006065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:26.006202 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:26.006633 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:26.006637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:26.006639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:26.007015 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:26.007328 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:26.007772 (+ 444us) server_negotiation.cc:299] Negotiation successful
0923 17:38:26.007956 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:38:26.008872 4989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:26.098356 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 4865
I20260923 17:38:26.111716 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 4504
I20260923 17:38:26.112950 4566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:26.225931 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 4504
I20260923 17:38:46.244166 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:35957
--webserver_interface=127.20.240.126
--webserver_port=37831
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:35957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:38:46.575001 5045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:46.575423 5045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:46.575575 5045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:46.583148 5045 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 17:38:46.583360 5045 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 17:38:46.583428 5045 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 17:38:46.583500 5045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:38:46.583576 5045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:46.583673 5045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:46.583738 5045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:38:46.583813 5045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:38:46.594373 5045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:35957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:35957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=37831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:46.595808 5045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:46.597148 5045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:38:46.620900 5045 server_base.cc:1034] running on GCE node
W20260923 17:38:46.620891 5052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:46.620909 5051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:46.621153 5054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:46.622305 5045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:46.624145 5045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:46.625483 5045 hybrid_clock.cc:648] HybridClock initialized: now 1790185126625394 us; error 60 us; skew 500 ppm
I20260923 17:38:46.625929 5045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:46 dist-test-slave-ppjl krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185126, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:46.647946 5045 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:38:46.649863 5045 webserver.cc:460] Webserver started at http://127.20.240.126:37831/ using document root <none> and password file <none>
I20260923 17:38:46.650710 5045 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:46.650846 5045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:46.657366 5045 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260923 17:38:46.660778 5061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:46.662446 5045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:38:46.662634 5045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "cd405a1902e64b24bbb1e81cc8ffbc43"
format_stamp: "Formatted at 2026-09-23 17:38:23 on dist-test-slave-ppjl"
server_key: "5be8f024338aca1b6c511f90e93e7fb4"
server_key_iv: "8f788692fb1fcb0f26ebf482200d5b3e"
server_key_version: "encryptionkey@0"
I20260923 17:38:46.663796 5045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:46.679991 5045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:46.684798 5045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:46.685101 5045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:46.703397 5045 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:35957
I20260923 17:38:46.703414 5112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:35957 every 8 connection(s)
I20260923 17:38:46.704938 5045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:38:46.708248 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 5045
I20260923 17:38:46.709230 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:34955
--local_ip_for_outbound_sockets=127.20.240.65
--tserver_master_addrs=127.20.240.126:35957
--webserver_port=37879
--webserver_interface=127.20.240.65
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:38:46.713794 5113 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:38:46.718566 5113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Bootstrap starting.
I20260923 17:38:46.738775 5113 log.cc:824] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:46.751332 5113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:38:46.751888 5113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Bootstrap complete.
I20260923 17:38:46.761077 5113 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } }
I20260923 17:38:46.762064 5113 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd405a1902e64b24bbb1e81cc8ffbc43, State: Initialized, Role: FOLLOWER
I20260923 17:38:46.762769 5113 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } }
I20260923 17:38:46.762989 5113 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:38:46.763111 5113 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:38:46.763310 5113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:38:46.767335 5113 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } }
I20260923 17:38:46.767892 5113 leader_election.cc:302] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd405a1902e64b24bbb1e81cc8ffbc43; no voters:
I20260923 17:38:46.768540 5113 leader_election.cc:288] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:38:46.768682 5117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:38:46.770742 5113 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:38:46.776206 5117 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 2 LEADER]: Becoming Leader. State: Replica: cd405a1902e64b24bbb1e81cc8ffbc43, State: Running, Role: LEADER
I20260923 17:38:46.776883 5117 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } }
I20260923 17:38:46.779541 5118 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } } }
I20260923 17:38:46.779994 5118 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [sys.catalog]: This master's current role is: LEADER
I20260923 17:38:46.780509 5119 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd405a1902e64b24bbb1e81cc8ffbc43. Latest consensus state: current_term: 2 leader_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } } }
I20260923 17:38:46.780788 5119 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [sys.catalog]: This master's current role is: LEADER
I20260923 17:38:46.782835 5126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:38:46.793834 5126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:38:46.795529 5126 catalog_manager.cc:1259] Loaded cluster ID: bd956bd3b52b4e00a09799c21c3c9b76
I20260923 17:38:46.795720 5126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:38:46.800290 5126 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:38:46.802160 5126 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Loaded TSK: 0
I20260923 17:38:46.860935 5126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Generated new TSK 1
I20260923 17:38:46.861377 5126 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:38:47.088649 5115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:47.088941 5115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:47.089054 5115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:47.096421 5115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:47.096546 5115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:47.096685 5115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:38:47.106606 5115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:34955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=37879
--tserver_master_addrs=127.20.240.126:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:47.107971 5115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:47.109278 5115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:47.124905 5144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:47.125280 5141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:47.124889 5142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:47.126479 5115 server_base.cc:1034] running on GCE node
I20260923 17:38:47.127439 5115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:47.128858 5115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:47.130131 5115 hybrid_clock.cc:648] HybridClock initialized: now 1790185127130072 us; error 43 us; skew 500 ppm
I20260923 17:38:47.130532 5115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:47 dist-test-slave-ppjl krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185127, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:47.134588 5115 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:38:47.136482 5115 webserver.cc:460] Webserver started at http://127.20.240.65:37879/ using document root <none> and password file <none>
I20260923 17:38:47.137444 5115 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:47.137611 5115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:47.145126 5115 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:38:47.148587 5151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:47.150231 5115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:38:47.150414 5115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "c19171edd1ba4b3d8dc6bca6775ea599"
format_stamp: "Formatted at 2026-09-23 17:38:24 on dist-test-slave-ppjl"
server_key: "dd5084ab16373d8c29dc9fb9490e41a0"
server_key_iv: "f15942c4199dd9882b016a99d0ec3699"
server_key_version: "encryptionkey@0"
I20260923 17:38:47.151831 5115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:47.168987 5115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:47.174485 5115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:47.174823 5115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:47.176172 5115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:38:47.177976 5115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:38:47.178097 5115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:47.178210 5115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:38:47.178287 5115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:47.213698 5115 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:34955
I20260923 17:38:47.213752 5263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:34955 every 8 connection(s)
I20260923 17:38:47.215649 5115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:38:47.216483 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 5115
I20260923 17:38:47.217408 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:42977
--local_ip_for_outbound_sockets=127.20.240.66
--tserver_master_addrs=127.20.240.126:35957
--webserver_port=32887
--webserver_interface=127.20.240.66
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:38:47 dist-test-slave-ppjl krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185127, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:38:47.247557 5266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:47.219126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:45921 (local address 127.20.240.126:35957)
0923 17:38:47.219877 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:47.219893 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:47.224144 (+ 4251us) server_negotiation.cc:407] Connection header received
0923 17:38:47.227048 (+ 2904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:47.227064 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:47.227583 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:47.227902 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:47.230123 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:47.231530 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:47.233953 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:47.234675 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:47.234829 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:47.239408 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:47.239419 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:47.239437 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:47.239477 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:47.242385 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:47.243519 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:47.243527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:47.243533 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:47.243729 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:47.244291 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:47.244296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:47.244298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:47.244867 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:47.245029 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:47.246108 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0923 17:38:47.246514 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":289,"threads_started":1}
I20260923 17:38:47.247867 5267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:47.219291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:45921)
0923 17:38:47.223863 (+ 4572us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:47.223941 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:47.226399 (+ 2458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:47.228174 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:47.228186 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:47.229008 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:38:47.229847 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:47.229880 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:47.231734 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:47.231740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:47.233708 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:47.233730 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:47.234951 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:47.234959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:47.235313 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:47.236066 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:38:47.236104 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:38:47.239053 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:38:47.242573 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:47.242585 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:47.242605 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:47.243337 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:47.243845 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:47.243851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:47.243856 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:47.244119 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:47.245010 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:38:47.245021 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:38:47.245472 (+ 451us) client_negotiation.cc:769] Sending connection context
0923 17:38:47.245867 (+ 395us) client_negotiation.cc:240] Negotiation successful
0923 17:38:47.246493 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4201,"thread_start_us":3921,"threads_started":1}
I20260923 17:38:47.250988 5264 heartbeater.cc:347] Connected to a master server at 127.20.240.126:35957
I20260923 17:38:47.251403 5264 heartbeater.cc:464] Registering TS with master...
I20260923 17:38:47.252328 5264 heartbeater.cc:511] Master 127.20.240.126:35957 requested a full tablet report, sending...
I20260923 17:38:47.254731 5078 ts_manager.cc:194] Registered new tserver with Master: c19171edd1ba4b3d8dc6bca6775ea599 (127.20.240.65:34955)
I20260923 17:38:47.256309 5078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:45921
W20260923 17:38:47.555701 5268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:47.556008 5268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:47.556134 5268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:47.563691 5268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:47.563823 5268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:47.563978 5268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:38:47.574333 5268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:42977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=32887
--tserver_master_addrs=127.20.240.126:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:47.575718 5268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:47.576987 5268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:47.589644 5273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:47.589721 5276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:47.589723 5274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:47.590792 5268 server_base.cc:1034] running on GCE node
I20260923 17:38:47.591467 5268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:47.592412 5268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:47.593629 5268 hybrid_clock.cc:648] HybridClock initialized: now 1790185127593584 us; error 31 us; skew 500 ppm
I20260923 17:38:47.593950 5268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:47 dist-test-slave-ppjl krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185127, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:47.596957 5268 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:38:47.598392 5268 webserver.cc:460] Webserver started at http://127.20.240.66:32887/ using document root <none> and password file <none>
I20260923 17:38:47.599082 5268 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:47.599236 5268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:47.604537 5268 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260923 17:38:47.607203 5283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:47.608724 5268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:38:47.608909 5268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "8b2354c55c5544a8bf4471195484345e"
format_stamp: "Formatted at 2026-09-23 17:38:24 on dist-test-slave-ppjl"
server_key: "041540b8cb8ca7e50b8ed085da785dc1"
server_key_iv: "e576e3bdb5de60796cb898e0e7144070"
server_key_version: "encryptionkey@0"
I20260923 17:38:47.609886 5268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:47.623668 5268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:47.628122 5268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:47.628432 5268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:47.629557 5268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:38:47.630976 5268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:38:47.631081 5268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:47.631197 5268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:38:47.631278 5268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260923 17:38:47.666272 5268 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:42977
I20260923 17:38:47.666381 5395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:42977 every 8 connection(s)
I20260923 17:38:47.667961 5268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:38:47.674173 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 5268
I20260923 17:38:47.674799 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:37297
--local_ip_for_outbound_sockets=127.20.240.67
--tserver_master_addrs=127.20.240.126:35957
--webserver_port=43989
--webserver_interface=127.20.240.67
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:38:47 dist-test-slave-ppjl krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185127, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:38:47.689059 5266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:47.670570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:40853 (local address 127.20.240.126:35957)
0923 17:38:47.670730 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:47.670735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:47.672110 (+ 1375us) server_negotiation.cc:407] Connection header received
0923 17:38:47.674083 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:47.674090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:47.674176 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:47.674264 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:47.675959 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:47.676651 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:47.678252 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:47.678669 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:47.678785 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:47.682710 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:47.682716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:47.682720 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:47.682772 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:47.685457 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:47.686488 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:47.686496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:47.686498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:47.686641 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:47.687208 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:47.687214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:47.687217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:47.687517 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:47.687658 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:47.688454 (+ 796us) server_negotiation.cc:299] Negotiation successful
0923 17:38:47.688709 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:38:47.690171 5398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:47.671100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:40853)
0923 17:38:47.671949 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:47.671987 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:47.673798 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:47.674538 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:47.674547 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:47.675137 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:38:47.675782 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:47.675800 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:47.676809 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:47.676812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:47.678095 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:47.678109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:47.678897 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:47.678904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:47.679151 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:47.679851 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:38:47.679874 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:38:47.682394 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:38:47.685638 (+ 3244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:47.685648 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:47.685668 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:47.686301 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:47.686814 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:47.686819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:47.686823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:47.687045 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:47.687649 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:38:47.687658 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:38:47.688083 (+ 425us) client_negotiation.cc:769] Sending connection context
0923 17:38:47.688332 (+ 249us) client_negotiation.cc:240] Negotiation successful
0923 17:38:47.688905 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":313,"threads_started":1}
I20260923 17:38:47.691906 5396 heartbeater.cc:347] Connected to a master server at 127.20.240.126:35957
I20260923 17:38:47.692222 5396 heartbeater.cc:464] Registering TS with master...
I20260923 17:38:47.693023 5396 heartbeater.cc:511] Master 127.20.240.126:35957 requested a full tablet report, sending...
I20260923 17:38:47.694494 5078 ts_manager.cc:194] Registered new tserver with Master: 8b2354c55c5544a8bf4471195484345e (127.20.240.66:42977)
I20260923 17:38:47.695312 5078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:40853
W20260923 17:38:48.009280 5399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:38:48.009618 5399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:38:48.009761 5399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:38:48.017318 5399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:38:48.017457 5399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:38:48.017621 5399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:38:48.027866 5399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:37297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=43989
--tserver_master_addrs=127.20.240.126:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:38:48.029269 5399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:38:48.030604 5399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:38:48.044806 5404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:38:48.044834 5405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:48.044991 5399 server_base.cc:1034] running on GCE node
W20260923 17:38:48.044822 5407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:38:48.046028 5399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:38:48.047088 5399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:38:48.048331 5399 hybrid_clock.cc:648] HybridClock initialized: now 1790185128048261 us; error 54 us; skew 500 ppm
I20260923 17:38:48.048664 5399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:38:48 dist-test-slave-ppjl krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185128, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:38:48.052282 5399 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:38:48.053833 5399 webserver.cc:460] Webserver started at http://127.20.240.67:43989/ using document root <none> and password file <none>
I20260923 17:38:48.054574 5399 fs_manager.cc:359] Metadata directory not provided
I20260923 17:38:48.054708 5399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:38:48.060837 5399 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260923 17:38:48.064345 5414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:48.066080 5399 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:38:48.066254 5399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "0461c3dc460541de93c36d9e0f788184"
format_stamp: "Formatted at 2026-09-23 17:38:25 on dist-test-slave-ppjl"
server_key: "4a5000bebe02edd07c57c04c7f526b81"
server_key_iv: "49b209950ed654a66ea9a2bc79e3f023"
server_key_version: "encryptionkey@0"
I20260923 17:38:48.067335 5399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:38:48.088722 5399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:38:48.093164 5399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:38:48.093398 5399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:38:48.094408 5399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:38:48.095777 5399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:38:48.095875 5399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:48.095957 5399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:38:48.096014 5399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:38:48.126673 5399 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:37297
I20260923 17:38:48.126747 5526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:37297 every 8 connection(s)
I20260923 17:38:48.128346 5399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:38:48.128680 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 5399
Sep 23 17:38:48 dist-test-slave-ppjl krb5kdc[4491](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185128, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:38:48.146520 5266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.129933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42136 (local address 127.20.240.126:35957)
0923 17:38:48.130138 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:48.130144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:48.130502 (+ 358us) server_negotiation.cc:407] Connection header received
0923 17:38:48.130718 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:48.130725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:48.130815 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:48.130931 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:48.132739 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.133566 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.135088 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.135743 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.135887 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.139554 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:48.139563 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:48.139567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:48.139634 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:48.143196 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:48.144065 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:48.144071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:48.144074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:48.144264 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:48.144734 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:48.144740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:48.144743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:48.145100 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:48.145359 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:48.145946 (+ 587us) server_negotiation.cc:299] Negotiation successful
0923 17:38:48.146132 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:38:48.153937 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.131412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:40597 (local address 127.20.240.126:35957)
0923 17:38:48.131886 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:48.131892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:48.132980 (+ 1088us) server_negotiation.cc:407] Connection header received
0923 17:38:48.134843 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:48.134850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:48.134934 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:48.135045 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:48.137530 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.138603 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.141361 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.142099 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.142236 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.146529 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:48.146536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:48.146541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:48.146578 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:48.149959 (+ 3381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:48.150938 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:48.150944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:48.150946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:48.151108 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:48.151783 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:48.151788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:48.151790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:48.152100 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:48.152226 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:48.153397 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0923 17:38:48.153601 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":248,"threads_started":1}
I20260923 17:38:48.154989 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.132006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:40597)
0923 17:38:48.132802 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:48.132863 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:48.134577 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:48.135365 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:48.135377 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:48.136226 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:38:48.137319 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:48.137345 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.138765 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.138771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:48.141079 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:48.141096 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.142446 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.142457 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:48.142779 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.143478 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:38:48.143506 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:38:48.146251 (+ 2745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:38:48.150165 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:48.150173 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:48.150191 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:48.150735 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:48.151293 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:38:48.151299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:38:48.151302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:38:48.151629 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:38:48.152409 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:38:48.152417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:38:48.152897 (+ 480us) client_negotiation.cc:769] Sending connection context
0923 17:38:48.153183 (+ 286us) client_negotiation.cc:240] Negotiation successful
0923 17:38:48.153649 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":313,"threads_started":1}
I20260923 17:38:48.156577 5527 heartbeater.cc:347] Connected to a master server at 127.20.240.126:35957
I20260923 17:38:48.156980 5527 heartbeater.cc:464] Registering TS with master...
I20260923 17:38:48.157845 5527 heartbeater.cc:511] Master 127.20.240.126:35957 requested a full tablet report, sending...
I20260923 17:38:48.159543 5078 ts_manager.cc:194] Registered new tserver with Master: 0461c3dc460541de93c36d9e0f788184 (127.20.240.67:37297)
I20260923 17:38:48.160499 5078 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:40597
I20260923 17:38:48.162281 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:38:48.169350 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.163417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.20.240.126:35957)
0923 17:38:48.163594 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:48.163599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:48.163984 (+ 385us) server_negotiation.cc:407] Connection header received
0923 17:38:48.164155 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:48.164159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:48.164356 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:48.164509 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:38:48.165012 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.165826 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.167404 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.167883 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.167991 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.168524 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:38:48.168764 (+ 240us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 17:38:48.169081 (+ 317us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42152: token expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260923 17:38:48.169473 5530 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42152: token expired
W20260923 17:38:48.170132 5000 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.240.126:35957)
I20260923 17:38:48.183507 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.170621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42164 (local address 127.20.240.126:35957)
0923 17:38:48.170755 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:48.170759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:48.170774 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:38:48.170920 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:48.170925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:48.170985 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:48.171075 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:48.172633 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.173569 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.174920 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.175381 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.175503 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.178109 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:48.178117 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:48.178121 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:48.178166 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:48.180932 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:48.181622 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:48.181627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:48.181630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:48.181779 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:48.182153 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:48.182156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:48.182158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:48.182539 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:48.182748 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:48.182994 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 17:38:48.183209 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 17:38:48.187373 5078 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:38:48.195051 5078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:38:48.229357 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.207594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.20.240.65:34955)
0923 17:38:48.214786 (+ 7192us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:48.214798 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:48.220142 (+ 5344us) server_negotiation.cc:407] Connection header received
0923 17:38:48.220621 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:48.220631 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:48.220912 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:48.221102 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:48.222252 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.223839 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.226317 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.227853 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.228076 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.228295 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:48.228811 (+ 516us) server_negotiation.cc:299] Negotiation successful
0923 17:38:48.229048 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6977,"thread_start_us":6799,"threads_started":1}
I20260923 17:38:48.229367 5537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.207275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34955 (local address 127.0.0.1:52762)
0923 17:38:48.219940 (+ 12665us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:48.219994 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:48.220329 (+ 335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:48.221541 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:48.221550 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:48.221590 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:48.222040 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:48.222054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.224003 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.224007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:48.226090 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:48.226102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.228437 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.228442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:48.228584 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.228624 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 17:38:48.228739 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 17:38:48.228951 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12374,"thread_start_us":12226,"threads_started":1}
I20260923 17:38:48.232291 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.208173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.20.240.67:37297)
0923 17:38:48.219021 (+ 10848us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:48.219032 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:48.223326 (+ 4294us) server_negotiation.cc:407] Connection header received
0923 17:38:48.223670 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:48.223679 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:48.223977 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:48.224155 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:48.225095 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.226726 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.229425 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.230852 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.230977 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.231232 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:48.231811 (+ 579us) server_negotiation.cc:299] Negotiation successful
0923 17:38:48.231983 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10608,"thread_start_us":10878,"threads_started":1}
I20260923 17:38:48.232781 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.220749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.20.240.66:42977)
0923 17:38:48.222638 (+ 1889us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:48.222649 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:48.223722 (+ 1073us) server_negotiation.cc:407] Connection header received
0923 17:38:48.223989 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:48.223997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:48.224214 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:48.224367 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:48.225379 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.226882 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.229426 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.231552 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.231695 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.231877 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:48.232286 (+ 409us) server_negotiation.cc:299] Negotiation successful
0923 17:38:48.232462 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1698,"thread_start_us":1564,"threads_started":1}
I20260923 17:38:48.233034 5542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.222707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:42977 (local address 127.0.0.1:53558)
0923 17:38:48.223616 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:48.223632 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:48.223817 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:48.224654 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:48.224660 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:48.224704 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:48.225192 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:48.225206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.227052 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.227058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:48.229254 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:48.229264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.231995 (+ 2731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.231999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:48.232117 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.232140 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:38:48.232220 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 17:38:48.232512 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":41,"thread_start_us":623,"threads_started":1}
I20260923 17:38:48.233500 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.207668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:37297 (local address 127.0.0.1:57504)
0923 17:38:48.223179 (+ 15511us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:48.223199 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:48.223456 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:48.224489 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:48.224493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:48.224518 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:48.224881 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:48.224894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.226900 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.226906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:48.229233 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:48.229249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.231391 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.231398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:48.231590 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.231623 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:38:48.232774 (+ 1151us) client_negotiation.cc:240] Negotiation successful
0923 17:38:48.232921 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15367,"mutex_wait_us":17,"thread_start_us":15207,"threads_started":1}
I20260923 17:38:48.233888 5199 tablet_service.cc:1467] Processing CreateTablet for tablet c1aa9be1c1494c788b606f761752981d (DEFAULT_TABLE table=test-table [id=11447bf3a9564ee593ec46eaf64e85db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:38:48.235565 5199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1aa9be1c1494c788b606f761752981d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:38:48.237231 5331 tablet_service.cc:1467] Processing CreateTablet for tablet c1aa9be1c1494c788b606f761752981d (DEFAULT_TABLE table=test-table [id=11447bf3a9564ee593ec46eaf64e85db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:38:48.237231 5462 tablet_service.cc:1467] Processing CreateTablet for tablet c1aa9be1c1494c788b606f761752981d (DEFAULT_TABLE table=test-table [id=11447bf3a9564ee593ec46eaf64e85db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:38:48.238867 5462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1aa9be1c1494c788b606f761752981d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:38:48.238857 5331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1aa9be1c1494c788b606f761752981d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:38:48.248793 5543 tablet_bootstrap.cc:492] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599: Bootstrap starting.
I20260923 17:38:48.252185 5543 tablet_bootstrap.cc:654] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599: Neither blocks nor log segments found. Creating new log.
I20260923 17:38:48.253268 5543 log.cc:824] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:48.255861 5545 tablet_bootstrap.cc:492] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e: Bootstrap starting.
I20260923 17:38:48.256575 5546 tablet_bootstrap.cc:492] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184: Bootstrap starting.
I20260923 17:38:48.258169 5543 tablet_bootstrap.cc:492] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599: No bootstrap required, opened a new log
I20260923 17:38:48.258463 5543 ts_tablet_manager.cc:1397] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260923 17:38:48.259560 5546 tablet_bootstrap.cc:654] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184: Neither blocks nor log segments found. Creating new log.
I20260923 17:38:48.259698 5545 tablet_bootstrap.cc:654] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e: Neither blocks nor log segments found. Creating new log.
I20260923 17:38:48.260146 5264 heartbeater.cc:503] Master 127.20.240.126:35957 was elected leader, sending a full tablet report...
I20260923 17:38:48.260840 5546 log.cc:824] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:48.260914 5545 log.cc:824] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:38:48.270228 5543 raft_consensus.cc:348] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } }
I20260923 17:38:48.270756 5543 raft_consensus.cc:374] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:38:48.270905 5543 raft_consensus.cc:729] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19171edd1ba4b3d8dc6bca6775ea599, State: Initialized, Role: FOLLOWER
I20260923 17:38:48.271036 5545 tablet_bootstrap.cc:492] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e: No bootstrap required, opened a new log
I20260923 17:38:48.271140 5546 tablet_bootstrap.cc:492] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184: No bootstrap required, opened a new log
I20260923 17:38:48.271464 5545 ts_tablet_manager.cc:1397] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20260923 17:38:48.271502 5546 ts_tablet_manager.cc:1397] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.002s
I20260923 17:38:48.271744 5543 consensus_queue.cc:260] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } }
I20260923 17:38:48.273742 5543 ts_tablet_manager.cc:1428] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260923 17:38:48.280186 5545 raft_consensus.cc:348] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } }
I20260923 17:38:48.280216 5546 raft_consensus.cc:348] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } }
I20260923 17:38:48.280591 5545 raft_consensus.cc:374] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:38:48.280598 5546 raft_consensus.cc:374] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:38:48.280687 5545 raft_consensus.cc:729] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2354c55c5544a8bf4471195484345e, State: Initialized, Role: FOLLOWER
I20260923 17:38:48.280691 5546 raft_consensus.cc:729] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0461c3dc460541de93c36d9e0f788184, State: Initialized, Role: FOLLOWER
I20260923 17:38:48.281237 5546 consensus_queue.cc:260] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } }
I20260923 17:38:48.281260 5545 consensus_queue.cc:260] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } }
I20260923 17:38:48.282418 5527 heartbeater.cc:503] Master 127.20.240.126:35957 was elected leader, sending a full tablet report...
I20260923 17:38:48.282835 5396 heartbeater.cc:503] Master 127.20.240.126:35957 was elected leader, sending a full tablet report...
I20260923 17:38:48.283325 5546 ts_tablet_manager.cc:1428] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260923 17:38:48.283565 5545 ts_tablet_manager.cc:1428] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260923 17:38:48.380191 5528 tablet.cc:2369] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:38:48.420181 5397 tablet.cc:2369] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:38:48.468966 5265 tablet.cc:2369] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:38:48.515483 5549 raft_consensus.cc:482] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:38:48.515779 5549 raft_consensus.cc:504] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } }
I20260923 17:38:48.517477 5549 leader_election.cc:288] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0461c3dc460541de93c36d9e0f788184 (127.20.240.67:37297), 8b2354c55c5544a8bf4471195484345e (127.20.240.66:42977)
I20260923 17:38:48.524570 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.517912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:36501 (local address 127.20.240.66:42977)
0923 17:38:48.518124 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:48.518130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:48.518984 (+ 854us) server_negotiation.cc:407] Connection header received
0923 17:38:48.519222 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:48.519227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:48.519299 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:48.519390 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:48.520235 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.521060 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.522886 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.523686 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.523838 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.523915 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:48.524175 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 17:38:48.524303 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:38:48.524684 5553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.517995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:42977 (local address 127.20.240.65:36501)
0923 17:38:48.518875 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:48.518891 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:48.519048 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:48.519558 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:48.519562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:48.519591 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:48.520076 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:48.520085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.521189 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.521194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:48.522726 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:48.522738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.523891 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.523897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:48.524027 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.524054 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:38:48.524138 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:38:48.524273 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":53,"thread_start_us":551,"threads_started":1}
I20260923 17:38:48.524833 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.518077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:57689 (local address 127.20.240.67:37297)
0923 17:38:48.518377 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:48.518383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:48.518803 (+ 420us) server_negotiation.cc:407] Connection header received
0923 17:38:48.518993 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:48.518999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:48.519095 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:48.519213 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:38:48.519969 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.520961 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.522997 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.523878 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.524050 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.524135 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:48.524432 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 17:38:48.524563 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 17:38:48.525094 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.517898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:37297 (local address 127.20.240.65:57689)
0923 17:38:48.518663 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:48.518681 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:38:48.518836 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:38:48.519227 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:38:48.519232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:38:48.519269 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:38:48.519801 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:48.519818 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.521117 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.521123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:48.522828 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:38:48.522841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.524105 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.524112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:38:48.524276 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.524302 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:38:48.524391 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 17:38:48.524528 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":57,"thread_start_us":392,"threads_started":1}
I20260923 17:38:48.525964 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1aa9be1c1494c788b606f761752981d" candidate_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2354c55c5544a8bf4471195484345e" is_pre_election: true
I20260923 17:38:48.526106 5482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1aa9be1c1494c788b606f761752981d" candidate_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0461c3dc460541de93c36d9e0f788184" is_pre_election: true
I20260923 17:38:48.526512 5351 raft_consensus.cc:2454] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c19171edd1ba4b3d8dc6bca6775ea599 in term 0.
I20260923 17:38:48.526557 5482 raft_consensus.cc:2454] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c19171edd1ba4b3d8dc6bca6775ea599 in term 0.
I20260923 17:38:48.527197 5155 leader_election.cc:302] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b2354c55c5544a8bf4471195484345e, c19171edd1ba4b3d8dc6bca6775ea599; no voters:
I20260923 17:38:48.527592 5549 raft_consensus.cc:2781] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:38:48.527750 5549 raft_consensus.cc:482] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:38:48.527859 5549 raft_consensus.cc:3037] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:38:48.530066 5549 raft_consensus.cc:504] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } }
I20260923 17:38:48.530710 5549 leader_election.cc:288] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [CANDIDATE]: Term 1 election: Requested vote from peers 0461c3dc460541de93c36d9e0f788184 (127.20.240.67:37297), 8b2354c55c5544a8bf4471195484345e (127.20.240.66:42977)
I20260923 17:38:48.531268 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1aa9be1c1494c788b606f761752981d" candidate_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b2354c55c5544a8bf4471195484345e"
I20260923 17:38:48.531208 5482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1aa9be1c1494c788b606f761752981d" candidate_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0461c3dc460541de93c36d9e0f788184"
I20260923 17:38:48.531478 5351 raft_consensus.cc:3037] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:38:48.531499 5482 raft_consensus.cc:3037] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:38:48.533736 5351 raft_consensus.cc:2454] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c19171edd1ba4b3d8dc6bca6775ea599 in term 1.
I20260923 17:38:48.533736 5482 raft_consensus.cc:2454] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c19171edd1ba4b3d8dc6bca6775ea599 in term 1.
I20260923 17:38:48.534200 5153 leader_election.cc:302] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0461c3dc460541de93c36d9e0f788184, c19171edd1ba4b3d8dc6bca6775ea599; no voters:
I20260923 17:38:48.534543 5549 raft_consensus.cc:2781] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:38:48.535022 5549 raft_consensus.cc:686] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [term 1 LEADER]: Becoming Leader. State: Replica: c19171edd1ba4b3d8dc6bca6775ea599, State: Running, Role: LEADER
I20260923 17:38:48.535689 5549 consensus_queue.cc:237] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } } peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } }
I20260923 17:38:48.540954 5077 catalog_manager.cc:5495] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 reported cstate change: term changed from 0 to 1, leader changed from <none> to c19171edd1ba4b3d8dc6bca6775ea599 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c19171edd1ba4b3d8dc6bca6775ea599" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 } health_report { overall_health: UNKNOWN } } }
I20260923 17:38:48.605116 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.599722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.20.240.65:34955)
0923 17:38:48.599931 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:48.599936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:48.599956 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:38:48.600056 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:48.600059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:48.600143 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:48.600297 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:38:48.600618 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.601428 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.602880 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.603394 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.603505 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.604022 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:38:48.604629 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:38:48.604747 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:48.604840 (+ 93us) server_negotiation.cc:299] Negotiation successful
0923 17:38:48.604914 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:38:48.613732 5482 raft_consensus.cc:1261] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184 [term 1 FOLLOWER]: Refusing update from remote peer c19171edd1ba4b3d8dc6bca6775ea599: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:38:48.613797 5351 raft_consensus.cc:1261] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e [term 1 FOLLOWER]: Refusing update from remote peer c19171edd1ba4b3d8dc6bca6775ea599: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:38:48.614841 5554 consensus_queue.cc:1035] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b2354c55c5544a8bf4471195484345e" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:38:48.615408 5549 consensus_queue.cc:1035] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0461c3dc460541de93c36d9e0f788184" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 37297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:38:48.626318 5557 mvcc.cc:205] Tried to move back new op lower bound from 7332598286791028736 to 7332598286486155264. Current Snapshot: MvccSnapshot[applied={T|T < 7332598286791028736}]
I20260923 17:38:48.633556 5558 mvcc.cc:205] Tried to move back new op lower bound from 7332598286791028736 to 7332598286486155264. Current Snapshot: MvccSnapshot[applied={T|T < 7332598286791028736}]
I20260923 17:38:48.642813 5077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:38:48.643123 5077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:38:48.647223 5077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Sending DeleteTablet for 3 replicas of tablet c1aa9be1c1494c788b606f761752981d
I20260923 17:38:48.649003 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 5115
I20260923 17:38:48.649485 5331 tablet_service.cc:1514] Processing DeleteTablet for tablet c1aa9be1c1494c788b606f761752981d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:38:48 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:53558
I20260923 17:38:48.650475 5462 tablet_service.cc:1514] Processing DeleteTablet for tablet c1aa9be1c1494c788b606f761752981d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:38:48 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:57504
I20260923 17:38:48.652560 5199 tablet_service.cc:1514] Processing DeleteTablet for tablet c1aa9be1c1494c788b606f761752981d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:38:48 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:52762
I20260923 17:38:48.664580 5568 tablet_replica.cc:330] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e: stopping tablet replica
I20260923 17:38:48.665585 5571 tablet_replica.cc:330] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599: stopping tablet replica
I20260923 17:38:48.666371 5571 raft_consensus.cc:2229] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:38:48.667101 5571 raft_consensus.cc:2258] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:38:48.667636 5568 raft_consensus.cc:2229] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:38:48.667902 5570 tablet_replica.cc:330] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184: stopping tablet replica
I20260923 17:38:48.668303 5570 raft_consensus.cc:2229] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:38:48.668818 5570 raft_consensus.cc:2258] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:38:48.668825 5568 raft_consensus.cc:2258] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:38:48.672359 5571 ts_tablet_manager.cc:1905] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:38:48.681028 5568 ts_tablet_manager.cc:1905] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:38:48.681027 5570 ts_tablet_manager.cc:1905] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:38:48.683718 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.650561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.20.240.65:34955)
0923 17:38:48.650745 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:48.650750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:48.660901 (+ 10151us) server_negotiation.cc:407] Connection header received
0923 17:38:48.661465 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:48.661472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:48.661571 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:48.661658 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:48.664859 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.666384 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.670506 (+ 4122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.671224 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.671330 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.675190 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:48.675208 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:48.675227 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:48.675280 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:48.679652 (+ 4372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:48.680729 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:48.680739 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:48.680743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:48.680953 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:48.681620 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:48.681624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:48.681628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:48.682118 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:48.682393 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:48.683045 (+ 652us) server_negotiation.cc:299] Negotiation successful
0923 17:38:48.683340 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:38:48.685773 5259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:48.688341 5568 ts_tablet_manager.cc:1918] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:38:48.688526 5568 log.cc:1196] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/c1aa9be1c1494c788b606f761752981d
I20260923 17:38:48.689069 5568 ts_tablet_manager.cc:1939] T c1aa9be1c1494c788b606f761752981d P 8b2354c55c5544a8bf4471195484345e: Deleting consensus metadata
I20260923 17:38:48.690258 5570 ts_tablet_manager.cc:1918] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:38:48.690438 5570 log.cc:1196] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/c1aa9be1c1494c788b606f761752981d
I20260923 17:38:48.690958 5570 ts_tablet_manager.cc:1939] T c1aa9be1c1494c788b606f761752981d P 0461c3dc460541de93c36d9e0f788184: Deleting consensus metadata
I20260923 17:38:48.691035 5065 catalog_manager.cc:4841] TS 8b2354c55c5544a8bf4471195484345e (127.20.240.66:42977): tablet c1aa9be1c1494c788b606f761752981d (table test-table [id=11447bf3a9564ee593ec46eaf64e85db]) successfully deleted
I20260923 17:38:48.693058 5063 catalog_manager.cc:4841] TS 0461c3dc460541de93c36d9e0f788184 (127.20.240.67:37297): tablet c1aa9be1c1494c788b606f761752981d (table test-table [id=11447bf3a9564ee593ec46eaf64e85db]) successfully deleted
I20260923 17:38:48.793443 5571 ts_tablet_manager.cc:1918] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:38:48.793848 5571 log.cc:1196] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/c1aa9be1c1494c788b606f761752981d
I20260923 17:38:48.794250 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 5115
W20260923 17:38:48.809154 5065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.240.65:34955 (error 108)
I20260923 17:38:48.809576 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 5268
I20260923 17:38:48.811507 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.810664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34955 (local address 127.0.0.1:52786)
0923 17:38:48.811129 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:48.811286 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":202,"threads_started":1}
W20260923 17:38:48.811759 5065 catalog_manager.cc:4584] TS c19171edd1ba4b3d8dc6bca6775ea599 (127.20.240.65:34955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c1aa9be1c1494c788b606f761752981d: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34955: connect: Connection refused (error 111)
I20260923 17:38:48.823766 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.810421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.20.240.66:42977)
0923 17:38:48.810624 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:48.810630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:48.810650 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:38:48.810760 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:48.810766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:48.810868 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:48.810981 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:48.812519 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.813238 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.814672 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.815097 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.815451 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.818025 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:48.818042 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:48.818060 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:48.818097 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:48.820942 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:48.821631 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:48.821637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:48.821640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:48.821776 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:48.822149 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:48.822152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:48.822154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:48.822548 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:48.822815 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:48.823152 (+ 337us) server_negotiation.cc:299] Negotiation successful
0923 17:38:48.823400 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:38:48.824405 5391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:48.865391 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.864909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34955 (local address 127.0.0.1:52794)
0923 17:38:48.865128 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:48.865219 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 17:38:48.866569 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.866129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34955 (local address 127.0.0.1:52802)
0923 17:38:48.866313 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:48.866426 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 17:38:48.915076 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 5268
I20260923 17:38:48.927300 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.926737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34955 (local address 127.0.0.1:52818)
0923 17:38:48.926982 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:48.927117 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260923 17:38:48.928444 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.928078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34955 (local address 127.0.0.1:52834)
0923 17:38:48.928225 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:48.928304 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 17:38:48.929600 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 5399
I20260923 17:38:48.944057 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:48.930274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.20.240.67:37297)
0923 17:38:48.930461 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:38:48.930466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:38:48.930484 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:38:48.930610 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:38:48.930614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:38:48.930676 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:38:48.930756 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:38:48.932157 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.932921 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.934358 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:38:48.934742 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:38:48.934832 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:38:48.937753 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:38:48.937771 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:38:48.937790 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:38:48.937834 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:38:48.940914 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:48.941816 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:48.941825 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:48.941827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:48.941973 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:38:48.942354 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:38:48.942358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:38:48.942360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:38:48.942746 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:38:48.943050 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0923 17:38:48.943487 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 17:38:48.943708 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:38:48.944765 5522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:49.008803 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:49.008249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34955 (local address 127.0.0.1:52850)
0923 17:38:49.008538 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:49.008631 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260923 17:38:49.010169 5574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:38:49.009710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34955 (local address 127.0.0.1:52856)
0923 17:38:49.009941 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 17:38:49.010024 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260923 17:38:49.033499 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 5399
I20260923 17:38:49.047338 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 5045
I20260923 17:38:49.048702 5108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:38:49.163548 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 5045
I20260923 17:39:09.183380 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:35957
--webserver_interface=127.20.240.126
--webserver_port=37831
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:35957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:09.527599 5578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:09.527927 5578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:09.528059 5578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:09.535454 5578 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 17:39:09.535563 5578 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 17:39:09.535606 5578 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 17:39:09.535681 5578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:39:09.535758 5578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:09.535843 5578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:09.535892 5578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:39:09.535967 5578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:39:09.546414 5578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:35957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:35957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=37831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:09.547808 5578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:09.549157 5578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:09.561161 5583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:09.561098 5586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:09.561132 5584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:09.562047 5578 server_base.cc:1034] running on GCE node
I20260923 17:39:09.562992 5578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:09.564651 5578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:09.565944 5578 hybrid_clock.cc:648] HybridClock initialized: now 1790185149565868 us; error 55 us; skew 500 ppm
I20260923 17:39:09.566358 5578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:09 dist-test-slave-ppjl krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185149, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:09.569646 5578 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:39:09.571382 5578 webserver.cc:460] Webserver started at http://127.20.240.126:37831/ using document root <none> and password file <none>
I20260923 17:39:09.572148 5578 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:09.572268 5578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:09.578254 5578 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260923 17:39:09.581316 5593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:09.582656 5578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:39:09.582854 5578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "cd405a1902e64b24bbb1e81cc8ffbc43"
format_stamp: "Formatted at 2026-09-23 17:38:23 on dist-test-slave-ppjl"
server_key: "5be8f024338aca1b6c511f90e93e7fb4"
server_key_iv: "8f788692fb1fcb0f26ebf482200d5b3e"
server_key_version: "encryptionkey@0"
I20260923 17:39:09.583875 5578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:09.604193 5578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:09.608659 5578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:09.608952 5578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:09.626792 5578 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:35957
I20260923 17:39:09.626827 5644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:35957 every 8 connection(s)
I20260923 17:39:09.628295 5578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:39:09.635507 5645 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:39:09.636788 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 5578
I20260923 17:39:09.637780 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:34955
--local_ip_for_outbound_sockets=127.20.240.65
--tserver_master_addrs=127.20.240.126:35957
--webserver_port=37879
--webserver_interface=127.20.240.65
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:39:09.640259 5645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Bootstrap starting.
I20260923 17:39:09.662827 5645 log.cc:824] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:09.680325 5645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:39:09.680882 5645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Bootstrap complete.
I20260923 17:39:09.690336 5645 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } }
I20260923 17:39:09.691320 5645 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd405a1902e64b24bbb1e81cc8ffbc43, State: Initialized, Role: FOLLOWER
I20260923 17:39:09.691954 5645 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } }
I20260923 17:39:09.692183 5645 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:39:09.692296 5645 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:39:09.692456 5645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 2 FOLLOWER]: Advancing to term 3
I20260923 17:39:09.696291 5645 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } }
I20260923 17:39:09.696789 5645 leader_election.cc:302] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd405a1902e64b24bbb1e81cc8ffbc43; no voters:
I20260923 17:39:09.697422 5645 leader_election.cc:288] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 17:39:09.697553 5649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 17:39:09.698587 5649 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [term 3 LEADER]: Becoming Leader. State: Replica: cd405a1902e64b24bbb1e81cc8ffbc43, State: Running, Role: LEADER
I20260923 17:39:09.699211 5649 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } }
I20260923 17:39:09.699560 5645 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:39:09.701613 5650 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } } }
I20260923 17:39:09.701987 5650 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:09.702580 5651 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd405a1902e64b24bbb1e81cc8ffbc43. Latest consensus state: current_term: 3 leader_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd405a1902e64b24bbb1e81cc8ffbc43" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35957 } } }
I20260923 17:39:09.702965 5651 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43 [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:09.704828 5654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:39:09.714360 5654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:39:09.716037 5654 catalog_manager.cc:1259] Loaded cluster ID: bd956bd3b52b4e00a09799c21c3c9b76
I20260923 17:39:09.716251 5654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:39:09.720438 5654 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:39:09.722177 5654 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Loaded TSK: 0
I20260923 17:39:09.722434 5654 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd405a1902e64b24bbb1e81cc8ffbc43: Loaded TSK: 1
F20260923 17:39:09.723749 5654 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f90ab68d9ad google::LogMessage::Fail() at ??:0
@ 0x7f90ab69088a google::LogMessage::SendToLog() at ??:0
@ 0x7f90ab68d4de google::LogMessage::Flush() at ??:0
@ 0x7f90ab68ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f90af3f17d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f90af3e073e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f90c10bc1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
I20260923 17:39:09.743528 5666 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f90c10bb783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f90c11472a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f90c11470c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f90c11915e9 std::function<>::operator()() at ??:0
@ 0x7f90c10bfcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f90c10bdc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f90c1145610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f90c1145439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f90ae2d9345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f90ae2dc91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f90ae2dc799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f90ae2b7eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f90b15916db start_thread at ??:0
@ 0x7f90aa52761f clone at ??:0
W20260923 17:39:10.028501 5647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:10.028806 5647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:10.028963 5647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:10.036485 5647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:10.036608 5647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:10.036753 5647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:39:10.046942 5647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:34955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=37879
--tserver_master_addrs=127.20.240.126:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:10.048327 5647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:10.049700 5647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:10.063102 5695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:10.063037 5692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:10.063127 5693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:10.063918 5647 server_base.cc:1034] running on GCE node
I20260923 17:39:10.064504 5647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:10.065488 5647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:10.066710 5647 hybrid_clock.cc:648] HybridClock initialized: now 1790185150066641 us; error 50 us; skew 500 ppm
I20260923 17:39:10.067072 5647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:10 dist-test-slave-ppjl krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185150, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:10.070314 5647 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:39:10.071890 5647 webserver.cc:460] Webserver started at http://127.20.240.65:37879/ using document root <none> and password file <none>
I20260923 17:39:10.072595 5647 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:10.072708 5647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:10.078928 5647 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:39:10.082108 5702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:10.083638 5647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:39:10.083798 5647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "c19171edd1ba4b3d8dc6bca6775ea599"
format_stamp: "Formatted at 2026-09-23 17:38:24 on dist-test-slave-ppjl"
server_key: "dd5084ab16373d8c29dc9fb9490e41a0"
server_key_iv: "f15942c4199dd9882b016a99d0ec3699"
server_key_version: "encryptionkey@0"
I20260923 17:39:10.084838 5647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:10.108299 5647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:10.112691 5647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:10.112957 5647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:10.114059 5647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:10.115932 5709 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260923 17:39:10.121340 5709 ts_tablet_manager.cc:1852] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260923 17:39:10.122331 5709 ts_tablet_manager.cc:1905] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:10.130038 5709 ts_tablet_manager.cc:1918] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:10.130332 5709 ts_tablet_manager.cc:1939] T c1aa9be1c1494c788b606f761752981d P c19171edd1ba4b3d8dc6bca6775ea599: Deleting consensus metadata
I20260923 17:39:10.130832 5647 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260923 17:39:10.130950 5647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260923 17:39:10.131074 5647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:10.131148 5647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:10.166548 5647 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:34955
I20260923 17:39:10.166595 5815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:34955 every 8 connection(s)
I20260923 17:39:10.167870 5647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:39:10.172611 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:10.170721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:33509)
0923 17:39:10.171482 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:10.171702 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":283,"threads_started":1}
W20260923 17:39:10.173221 5816 heartbeater.cc:644] Failed to heartbeat to 127.20.240.126:35957 (0 consecutive failures): Network error: Failed to ping master at 127.20.240.126:35957: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
I20260923 17:39:10.174089 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:10.173725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:44905)
0923 17:39:10.173900 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:10.173975 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 17:39:10.175036 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:10.174646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:53067)
0923 17:39:10.174818 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:10.174904 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260923 17:39:10.175335 5816 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:39:10.175537 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 5647
I20260923 17:39:10.176129 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:42977
--local_ip_for_outbound_sockets=127.20.240.66
--tserver_master_addrs=127.20.240.126:35957
--webserver_port=32887
--webserver_interface=127.20.240.66
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:10.502161 5819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:10.502468 5819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:10.502563 5819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:10.509837 5819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:10.509960 5819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:10.510076 5819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:39:10.520109 5819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:42977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=32887
--tserver_master_addrs=127.20.240.126:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:10.521435 5819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:10.522871 5819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:10.537025 5824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:10.537002 5827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:10.536973 5825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:10.537941 5819 server_base.cc:1034] running on GCE node
I20260923 17:39:10.538589 5819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:10.539647 5819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:10.540860 5819 hybrid_clock.cc:648] HybridClock initialized: now 1790185150540814 us; error 34 us; skew 500 ppm
I20260923 17:39:10.541183 5819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:10 dist-test-slave-ppjl krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185150, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:10.544348 5819 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:39:10.545899 5819 webserver.cc:460] Webserver started at http://127.20.240.66:32887/ using document root <none> and password file <none>
I20260923 17:39:10.546593 5819 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:10.546720 5819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:10.552500 5819 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260923 17:39:10.555579 5834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:10.557101 5819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 17:39:10.557287 5819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "8b2354c55c5544a8bf4471195484345e"
format_stamp: "Formatted at 2026-09-23 17:38:24 on dist-test-slave-ppjl"
server_key: "041540b8cb8ca7e50b8ed085da785dc1"
server_key_iv: "e576e3bdb5de60796cb898e0e7144070"
server_key_version: "encryptionkey@0"
I20260923 17:39:10.558321 5819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:10.573125 5819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:10.577302 5819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:10.577570 5819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:10.578591 5819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:10.580008 5819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:10.580103 5819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:10.580207 5819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:10.580286 5819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:10.611248 5819 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:42977
I20260923 17:39:10.611321 5946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:42977 every 8 connection(s)
I20260923 17:39:10.612699 5819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:39:10.617847 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:10.615839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:57665)
0923 17:39:10.616600 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:10.616834 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":294,"threads_started":1}
W20260923 17:39:10.618489 5947 heartbeater.cc:644] Failed to heartbeat to 127.20.240.126:35957 (0 consecutive failures): Network error: Failed to ping master at 127.20.240.126:35957: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
I20260923 17:39:10.619319 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 5819
I20260923 17:39:10.619431 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:10.618989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:53463)
0923 17:39:10.619198 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:10.619290 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260923 17:39:10.620014 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:37297
--local_ip_for_outbound_sockets=127.20.240.67
--tserver_master_addrs=127.20.240.126:35957
--webserver_port=43989
--webserver_interface=127.20.240.67
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:39:10.620353 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:10.619960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:59857)
0923 17:39:10.620117 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:10.620216 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 17:39:10.620607 5947 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:39:10.952093 5950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:10.952422 5950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:10.952571 5950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:10.959991 5950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:10.960141 5950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:10.960299 5950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:39:10.970566 5950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:37297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=43989
--tserver_master_addrs=127.20.240.126:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:10.972070 5950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:10.973461 5950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:10.986646 5956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:10.986673 5955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:10.986665 5958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:10.987668 5950 server_base.cc:1034] running on GCE node
I20260923 17:39:10.988317 5950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:10.989372 5950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:10.990592 5950 hybrid_clock.cc:648] HybridClock initialized: now 1790185150990529 us; error 55 us; skew 500 ppm
I20260923 17:39:10.990931 5950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:10 dist-test-slave-ppjl krb5kdc[4491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185150, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:10.994412 5950 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:39:10.996024 5950 webserver.cc:460] Webserver started at http://127.20.240.67:43989/ using document root <none> and password file <none>
I20260923 17:39:10.996726 5950 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:10.996847 5950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:11.002801 5950 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:39:11.005687 5965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:11.007103 5950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:39:11.007314 5950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "0461c3dc460541de93c36d9e0f788184"
format_stamp: "Formatted at 2026-09-23 17:38:25 on dist-test-slave-ppjl"
server_key: "4a5000bebe02edd07c57c04c7f526b81"
server_key_iv: "49b209950ed654a66ea9a2bc79e3f023"
server_key_version: "encryptionkey@0"
I20260923 17:39:11.008325 5950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:11.024639 5950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:11.029042 5950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:11.029304 5950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:11.030361 5950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:11.031821 5950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:11.031926 5950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:11.032037 5950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:11.032114 5950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:11.062412 5950 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:37297
I20260923 17:39:11.062521 6077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:37297 every 8 connection(s)
I20260923 17:39:11.064085 5950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:39:11.069175 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:11.067199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:34385)
0923 17:39:11.068020 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:11.068266 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":296,"threads_started":1}
W20260923 17:39:11.069967 6078 heartbeater.cc:644] Failed to heartbeat to 127.20.240.126:35957 (0 consecutive failures): Network error: Failed to ping master at 127.20.240.126:35957: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
I20260923 17:39:11.070837 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:11.070487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:48295)
0923 17:39:11.070627 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:11.070729 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 17:39:11.071619 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:11.071329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:44429)
0923 17:39:11.071440 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:11.071512 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 17:39:11.071856 6078 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:39:11.073796 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 5950
I20260923 17:39:11.176759 6082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:11.175912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:48853)
0923 17:39:11.176376 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:11.176467 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":228,"threads_started":1}
I20260923 17:39:11.621918 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:11.621115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:38919)
0923 17:39:11.621561 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:11.621662 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":228,"threads_started":1}
I20260923 17:39:12.073262 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:12.072418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:52359)
0923 17:39:12.072874 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:12.072983 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":216,"threads_started":1}
I20260923 17:39:12.178545 6085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:12.177799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:40455)
0923 17:39:12.178278 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:12.178373 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":221,"threads_started":1}
I20260923 17:39:12.623711 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:12.622927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:49885)
0923 17:39:12.623456 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:12.623538 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":291,"threads_started":1}
I20260923 17:39:13.075130 6087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:13.074317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:41927)
0923 17:39:13.074862 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:13.074956 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":265,"threads_started":1}
I20260923 17:39:13.180380 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:13.179589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:45795)
0923 17:39:13.180074 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:13.180173 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":235,"threads_started":1}
I20260923 17:39:13.625404 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:13.624662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:43829)
0923 17:39:13.625111 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:13.625217 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":231,"threads_started":1}
I20260923 17:39:14.076864 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:14.076121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:56767)
0923 17:39:14.076601 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:14.076688 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":252,"threads_started":1}
I20260923 17:39:14.182113 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:14.181315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:44675)
0923 17:39:14.181840 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:14.181932 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":251,"threads_started":1}
I20260923 17:39:14.627282 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:14.626357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:42371)
0923 17:39:14.626903 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:14.626995 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":248,"threads_started":1}
I20260923 17:39:15.078882 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:15.077982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:39613)
0923 17:39:15.078541 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:15.078642 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":269,"threads_started":1}
I20260923 17:39:15.183920 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:15.183149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:39017)
0923 17:39:15.183679 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:15.183763 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":253,"threads_started":1}
I20260923 17:39:15.629036 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:15.628253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:37961)
0923 17:39:15.628752 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:15.628842 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":243,"threads_started":1}
I20260923 17:39:16.080811 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:16.079961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:46771)
0923 17:39:16.080542 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:16.080635 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":322,"threads_started":1}
I20260923 17:39:16.185770 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:16.184992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:52475)
0923 17:39:16.185469 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:16.185569 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":227,"threads_started":1}
I20260923 17:39:16.630776 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:16.629980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:36549)
0923 17:39:16.630496 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:16.630592 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":274,"threads_started":1}
I20260923 17:39:17.082679 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:17.081863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:49673)
0923 17:39:17.082374 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:17.082484 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":272,"threads_started":1}
I20260923 17:39:17.187683 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:17.186686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:44765)
0923 17:39:17.187364 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:17.187460 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":337,"threads_started":1}
I20260923 17:39:17.632696 6101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:17.631739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:59729)
0923 17:39:17.632327 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:17.632424 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":263,"threads_started":1}
I20260923 17:39:18.084599 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:18.083749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:59941)
0923 17:39:18.084293 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:18.084383 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":246,"threads_started":1}
I20260923 17:39:18.189427 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:18.188592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:51893)
0923 17:39:18.189123 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:18.189216 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":271,"threads_started":1}
I20260923 17:39:18.634490 6104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:18.633739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:42705)
0923 17:39:18.634221 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:18.634314 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":238,"threads_started":1}
I20260923 17:39:19.086329 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:19.085615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:59397)
0923 17:39:19.086070 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:19.086162 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":228,"threads_started":1}
I20260923 17:39:19.191044 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:19.190323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:35651)
0923 17:39:19.190785 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:19.190883 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":212,"threads_started":1}
I20260923 17:39:19.636188 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:19.635410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:38167)
0923 17:39:19.635926 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:19.636009 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":277,"threads_started":1}
I20260923 17:39:20.088146 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:20.087382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:58867)
0923 17:39:20.087876 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:20.087968 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":224,"threads_started":1}
I20260923 17:39:20.192775 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:20.192061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:44135)
0923 17:39:20.192505 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:20.192594 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":226,"threads_started":1}
I20260923 17:39:20.637851 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:20.637143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:38887)
0923 17:39:20.637572 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:20.637681 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":228,"threads_started":1}
I20260923 17:39:21.090037 6111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:21.089243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:35869)
0923 17:39:21.089791 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:21.089873 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":339,"threads_started":1}
I20260923 17:39:21.194432 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:21.193680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:56729)
0923 17:39:21.194120 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:21.194261 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":239,"threads_started":1}
I20260923 17:39:21.639647 6113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:21.638871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:58911)
0923 17:39:21.639352 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:21.639452 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":211,"threads_started":1}
I20260923 17:39:22.091848 6114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:22.091081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:41157)
0923 17:39:22.091555 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:22.091661 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":231,"threads_started":1}
I20260923 17:39:22.196209 6115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:22.195442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:41827)
0923 17:39:22.195909 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:22.195996 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":271,"threads_started":1}
I20260923 17:39:22.641393 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:22.640689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:56493)
0923 17:39:22.641143 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:22.641233 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":217,"threads_started":1}
I20260923 17:39:23.093477 6117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:23.092821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:40923)
0923 17:39:23.093237 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:23.093322 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":201,"threads_started":1}
I20260923 17:39:23.198122 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:23.197326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:54951)
0923 17:39:23.197840 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:23.197945 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":235,"threads_started":1}
I20260923 17:39:23.643122 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:23.642355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:52651)
0923 17:39:23.642835 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:23.642935 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":235,"threads_started":1}
I20260923 17:39:24.095245 6120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:24.094409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:38961)
0923 17:39:24.094901 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:24.095002 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":249,"threads_started":1}
I20260923 17:39:24.200057 6121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:24.199145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:43967)
0923 17:39:24.199683 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:24.199790 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":226,"threads_started":1}
I20260923 17:39:24.644735 6122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:24.644036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:54977)
0923 17:39:24.644456 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:24.644551 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":212,"threads_started":1}
I20260923 17:39:25.097050 6123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:25.096130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:40693)
0923 17:39:25.096636 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:25.096746 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":247,"threads_started":1}
I20260923 17:39:25.202229 6124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:25.201367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.65:42163)
0923 17:39:25.201845 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:25.202047 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":224,"threads_started":1}
I20260923 17:39:25.646724 6125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:25.645886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.66:39985)
0923 17:39:25.646395 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:25.646535 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":273,"threads_started":1}
I20260923 17:39:26.079931 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 5647
I20260923 17:39:26.099287 6127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:26.098247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35957 (local address 127.20.240.67:48369)
0923 17:39:26.098764 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:26.098888 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.126:35957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":277,"threads_started":1}
I20260923 17:39:26.104001 6126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:26.081114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.20.240.65:34955)
0923 17:39:26.081741 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:26.081760 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:26.081820 (+ 60us) server_negotiation.cc:407] Connection header received
0923 17:39:26.083583 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:26.083594 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:26.084056 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:26.084402 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:26.085979 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:26.087282 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:26.088616 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:26.089393 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:26.089587 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:26.097023 (+ 7436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:26.097035 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:26.097055 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:26.097119 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:26.100640 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:26.101412 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:26.101420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:26.101429 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:26.101626 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:26.101967 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:26.101970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:26.101973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:26.102575 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:26.102832 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:26.103291 (+ 459us) server_negotiation.cc:299] Negotiation successful
0923 17:39:26.103705 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":242,"threads_started":1}
I20260923 17:39:26.105376 5811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:26.196406 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 5647
I20260923 17:39:26.211766 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 5819
I20260923 17:39:26.229820 6129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:26.212721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.20.240.66:42977)
0923 17:39:26.213273 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:26.213286 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:26.213323 (+ 37us) server_negotiation.cc:407] Connection header received
0923 17:39:26.214900 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:26.214910 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:26.215392 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:26.215687 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:26.217039 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:26.218304 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:26.219679 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:26.220367 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:26.220512 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:26.223152 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:26.223266 (+ 114us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:26.223285 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:26.223330 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:26.226346 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:26.227196 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:26.227205 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:26.227215 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:26.227402 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:26.227784 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:26.227787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:26.227789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:26.228435 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:26.228679 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:26.229127 (+ 448us) server_negotiation.cc:299] Negotiation successful
0923 17:39:26.229537 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":267,"threads_started":1}
I20260923 17:39:26.231389 5942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:26.315801 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 5819
I20260923 17:39:26.330614 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 5950
I20260923 17:39:26.349133 6131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:26.331633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.20.240.67:37297)
0923 17:39:26.332249 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:26.332263 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:26.332306 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:39:26.333925 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:26.333936 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:26.334375 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:26.334711 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:26.336345 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:26.337643 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:26.338960 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:26.339719 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:26.339903 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:26.342735 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:26.342744 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:26.342761 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:26.342804 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:26.345693 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:26.346421 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:26.346428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:26.346435 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:26.346644 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:26.347021 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:26.347025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:26.347027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:26.347649 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:26.347948 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:26.348385 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 17:39:26.348824 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":262,"threads_started":1}
I20260923 17:39:26.350631 6073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:26.435542 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 5950
2026-09-23T17:39:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64626 ms)
[----------] 30 tests from SecurityITest (300419 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:39:26 dist-test-slave-ppjl krb5kdc[6135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:39:26 dist-test-slave-ppjl krb5kdc[6135](info): set up 2 sockets
Sep 23 17:39:26 dist-test-slave-ppjl krb5kdc[6135](info): commencing operation
krb5kdc: starting...
W20260923 17:39:27.467921 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:39:27 dist-test-slave-ppjl krb5kdc[6135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:39:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:39:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:39:27.715529 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:39185
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:39185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:28.047271 6148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:28.047621 6148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:28.047729 6148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:28.055526 6148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:39:28.055670 6148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:28.055723 6148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:28.055787 6148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:39:28.055837 6148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:39:28.066771 6148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:39185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:39185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:28.068285 6148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:28.069692 6148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:28.082798 6154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:28.082746 6156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:28.082834 6153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:28.083329 6148 server_base.cc:1034] running on GCE node
I20260923 17:39:28.084125 6148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:28.085731 6148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:28.086989 6148 hybrid_clock.cc:648] HybridClock initialized: now 1790185168086907 us; error 60 us; skew 500 ppm
I20260923 17:39:28.087416 6148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:28 dist-test-slave-ppjl krb5kdc[6135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:28.090585 6148 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:39:28.092218 6148 webserver.cc:460] Webserver started at http://127.20.240.126:35815/ using document root <none> and password file <none>
I20260923 17:39:28.092926 6148 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:28.093050 6148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:28.093339 6148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:28.096417 6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "34316b9d0110472ea6a380a0fe1b7513"
format_stamp: "Formatted at 2026-09-23 17:39:28 on dist-test-slave-ppjl"
server_key: "3dc985320a03242652c28264123205a3"
server_key_iv: "a69e06cf070b47f43f24bc04d7fa571d"
server_key_version: "encryptionkey@0"
I20260923 17:39:28.097220 6148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "34316b9d0110472ea6a380a0fe1b7513"
format_stamp: "Formatted at 2026-09-23 17:39:28 on dist-test-slave-ppjl"
server_key: "3dc985320a03242652c28264123205a3"
server_key_iv: "a69e06cf070b47f43f24bc04d7fa571d"
server_key_version: "encryptionkey@0"
I20260923 17:39:28.102893 6148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:39:28.106766 6163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:28.108407 6148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:39:28.108613 6148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "34316b9d0110472ea6a380a0fe1b7513"
format_stamp: "Formatted at 2026-09-23 17:39:28 on dist-test-slave-ppjl"
server_key: "3dc985320a03242652c28264123205a3"
server_key_iv: "a69e06cf070b47f43f24bc04d7fa571d"
server_key_version: "encryptionkey@0"
I20260923 17:39:28.108821 6148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:28.122067 6148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:28.126190 6148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:28.126463 6148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:28.146059 6148 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:39185
I20260923 17:39:28.146068 6214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:39185 every 8 connection(s)
I20260923 17:39:28.147502 6148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:39:28.149197 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 6148
I20260923 17:39:28.149384 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:39:28.150015 21441 external_mini_cluster.cc:1428] Setting key 17e3af1820290e0c78e8a84e38182f89
I20260923 17:39:28.151834 6215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:39:28 dist-test-slave-ppjl krb5kdc[6135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:39:28.165944 6215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513: Bootstrap starting.
I20260923 17:39:28.169503 6215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:28.170856 6215 log.cc:824] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:28.173745 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:28.151924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.20.240.126:39185)
0923 17:39:28.152702 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:28.152714 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:28.152757 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:39:28.154385 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:28.154410 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:28.154848 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:28.155241 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:28.157077 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:28.158485 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:28.160408 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:28.161142 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:28.161285 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:28.166353 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:28.166361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:28.166376 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:28.166419 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:28.169372 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:28.170262 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:28.170270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:28.170277 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:28.170447 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:28.170866 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:28.170869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:28.170872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:28.171547 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:28.171852 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:28.172290 (+ 438us) server_negotiation.cc:299] Negotiation successful
0923 17:39:28.172720 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":321,"threads_started":1}
I20260923 17:39:28.174046 6215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513: No bootstrap required, opened a new log
I20260923 17:39:28.181659 6215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34316b9d0110472ea6a380a0fe1b7513" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39185 } }
I20260923 17:39:28.182067 6215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:28.182180 6215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34316b9d0110472ea6a380a0fe1b7513, State: Initialized, Role: FOLLOWER
I20260923 17:39:28.182823 6215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34316b9d0110472ea6a380a0fe1b7513" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39185 } }
I20260923 17:39:28.183104 6215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:39:28.183302 6215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:39:28.183465 6215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:28.185698 6215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34316b9d0110472ea6a380a0fe1b7513" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39185 } }
I20260923 17:39:28.186203 6215 leader_election.cc:302] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34316b9d0110472ea6a380a0fe1b7513; no voters:
I20260923 17:39:28.186772 6215 leader_election.cc:288] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:39:28.186898 6220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:28.187564 6220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [term 1 LEADER]: Becoming Leader. State: Replica: 34316b9d0110472ea6a380a0fe1b7513, State: Running, Role: LEADER
I20260923 17:39:28.188097 6220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34316b9d0110472ea6a380a0fe1b7513" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39185 } }
I20260923 17:39:28.188788 6215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:39:28.190819 6222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34316b9d0110472ea6a380a0fe1b7513. Latest consensus state: current_term: 1 leader_uuid: "34316b9d0110472ea6a380a0fe1b7513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34316b9d0110472ea6a380a0fe1b7513" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39185 } } }
I20260923 17:39:28.190828 6221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34316b9d0110472ea6a380a0fe1b7513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34316b9d0110472ea6a380a0fe1b7513" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 39185 } } }
I20260923 17:39:28.191280 6221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:28.191280 6222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513 [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:28.192188 6225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:39:28.197289 6225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:39:28.208603 6225 catalog_manager.cc:1347] Generated new cluster ID: 9eb08f1524924ff3b6c4d51b28af42c8
I20260923 17:39:28.208737 6225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:39:28.221288 6225 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:39:28.223075 6225 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:39:28.235260 6225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513: Generated new TSK 0
I20260923 17:39:28.236138 6225 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:39:28.306515 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:39185
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:28.663645 6241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:28.664100 6241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:28.664224 6241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:28.671942 6241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:28.672070 6241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:28.672201 6241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:39:28.682638 6241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:28.684048 6241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:28.685343 6241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:28.699105 6249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:28.699185 6246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:28.699247 6247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:28.700055 6241 server_base.cc:1034] running on GCE node
I20260923 17:39:28.700742 6241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:28.701788 6241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:28.703001 6241 hybrid_clock.cc:648] HybridClock initialized: now 1790185168702943 us; error 46 us; skew 500 ppm
I20260923 17:39:28.703430 6241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:28 dist-test-slave-ppjl krb5kdc[6135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:28.706828 6241 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:39:28.708388 6241 webserver.cc:460] Webserver started at http://127.20.240.65:45113/ using document root <none> and password file <none>
I20260923 17:39:28.709115 6241 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:28.709231 6241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:28.709545 6241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:28.712638 6241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb"
format_stamp: "Formatted at 2026-09-23 17:39:28 on dist-test-slave-ppjl"
server_key: "d8dcb251f29fd33acc44305674699270"
server_key_iv: "3b35c281be025a32c7fd4e0f20e74b0b"
server_key_version: "encryptionkey@0"
I20260923 17:39:28.713452 6241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb"
format_stamp: "Formatted at 2026-09-23 17:39:28 on dist-test-slave-ppjl"
server_key: "d8dcb251f29fd33acc44305674699270"
server_key_iv: "3b35c281be025a32c7fd4e0f20e74b0b"
server_key_version: "encryptionkey@0"
I20260923 17:39:28.718995 6241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:39:28.722729 6256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:28.724184 6241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:39:28.724349 6241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb"
format_stamp: "Formatted at 2026-09-23 17:39:28 on dist-test-slave-ppjl"
server_key: "d8dcb251f29fd33acc44305674699270"
server_key_iv: "3b35c281be025a32c7fd4e0f20e74b0b"
server_key_version: "encryptionkey@0"
I20260923 17:39:28.724541 6241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:28.739223 6241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:28.743628 6241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:28.743911 6241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:28.744894 6241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:28.746321 6241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:28.746405 6241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:28.746531 6241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:28.746588 6241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:28.777263 6241 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:43963
I20260923 17:39:28.777307 6368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:43963 every 8 connection(s)
I20260923 17:39:28.778556 6241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:39:28.785172 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 6241
I20260923 17:39:28.785401 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:39:28.786243 21441 external_mini_cluster.cc:1428] Setting key f2f6987bd8b5f910e66e1a7c5e43b85a
Sep 23 17:39:28 dist-test-slave-ppjl krb5kdc[6135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185168, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:39:28.801040 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:28.781216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:37533 (local address 127.20.240.126:39185)
0923 17:39:28.781701 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:28.781708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:28.782693 (+ 985us) server_negotiation.cc:407] Connection header received
0923 17:39:28.784597 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:28.784602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:28.784691 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:28.784787 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:28.787063 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:28.788083 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:28.790382 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:28.790805 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:28.790916 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:28.795090 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:28.795096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:28.795099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:28.795134 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:28.797504 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:28.798423 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:28.798429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:28.798432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:28.798573 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:28.799147 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:28.799176 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:28.799180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:28.799485 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:28.799638 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:28.800398 (+ 760us) server_negotiation.cc:299] Negotiation successful
0923 17:39:28.800645 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":228,"threads_started":1}
I20260923 17:39:28.802078 6372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:28.781772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:39185 (local address 127.20.240.65:37533)
0923 17:39:28.782502 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:28.782556 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:28.784325 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:28.785175 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:28.785187 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:28.785958 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:28.786817 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:28.786845 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:28.788266 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:28.788272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:28.790222 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:28.790238 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:28.791045 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:28.791051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:28.791374 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:28.792089 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:39:28.792119 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:28.794806 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:28.797664 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:28.797674 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:28.797694 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:28.798312 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:28.798700 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:28.798703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:28.798706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:28.798906 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:28.799638 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:28.799646 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:28.799990 (+ 344us) client_negotiation.cc:769] Sending connection context
0923 17:39:28.800295 (+ 305us) client_negotiation.cc:240] Negotiation successful
0923 17:39:28.800875 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":282,"threads_started":1}
I20260923 17:39:28.803833 6369 heartbeater.cc:347] Connected to a master server at 127.20.240.126:39185
I20260923 17:39:28.804164 6369 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:28.804905 6369 heartbeater.cc:511] Master 127.20.240.126:39185 requested a full tablet report, sending...
I20260923 17:39:28.807473 6180 ts_manager.cc:194] Registered new tserver with Master: 0b57b3efbbde4d5ebb20d5ccbb4b86fb (127.20.240.65:43963)
I20260923 17:39:28.809671 6180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:37533
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:39:28.842942 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:39185
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:29.174764 6375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:29.175038 6375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:29.175218 6375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:29.182464 6375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:29.182582 6375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:29.182696 6375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:39:29.192548 6375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:29.193809 6375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:29.195073 6375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:29.208123 6383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:29.208309 6380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:29.208678 6381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:29.208976 6375 server_base.cc:1034] running on GCE node
I20260923 17:39:29.209535 6375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:29.210433 6375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:29.211643 6375 hybrid_clock.cc:648] HybridClock initialized: now 1790185169211590 us; error 46 us; skew 500 ppm
I20260923 17:39:29.211966 6375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:29 dist-test-slave-ppjl krb5kdc[6135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:29.215137 6375 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:39:29.216604 6375 webserver.cc:460] Webserver started at http://127.20.240.66:39455/ using document root <none> and password file <none>
I20260923 17:39:29.217285 6375 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:29.217399 6375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:29.217713 6375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:29.220826 6375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "a145f8d4584445e0935e565fed5651f1"
format_stamp: "Formatted at 2026-09-23 17:39:29 on dist-test-slave-ppjl"
server_key: "157abe6c3672f6bf9151db227184856f"
server_key_iv: "0f4e4c996f83d3e8760693bb275183b7"
server_key_version: "encryptionkey@0"
I20260923 17:39:29.221637 6375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "a145f8d4584445e0935e565fed5651f1"
format_stamp: "Formatted at 2026-09-23 17:39:29 on dist-test-slave-ppjl"
server_key: "157abe6c3672f6bf9151db227184856f"
server_key_iv: "0f4e4c996f83d3e8760693bb275183b7"
server_key_version: "encryptionkey@0"
I20260923 17:39:29.227021 6375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:39:29.230710 6390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:29.232265 6375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:39:29.232466 6375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "a145f8d4584445e0935e565fed5651f1"
format_stamp: "Formatted at 2026-09-23 17:39:29 on dist-test-slave-ppjl"
server_key: "157abe6c3672f6bf9151db227184856f"
server_key_iv: "0f4e4c996f83d3e8760693bb275183b7"
server_key_version: "encryptionkey@0"
I20260923 17:39:29.232686 6375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:29.257215 6375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:29.261548 6375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:29.261822 6375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:29.262822 6375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:29.264353 6375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:29.264430 6375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:29.264529 6375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:29.264580 6375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:29.295727 6375 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:45431
I20260923 17:39:29.295776 6502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:45431 every 8 connection(s)
I20260923 17:39:29.296988 6375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:39:29.306610 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 6375
I20260923 17:39:29.306771 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:39:29.307247 21441 external_mini_cluster.cc:1428] Setting key 3f5094461c58dc95bb7bf1085baeaf45
Sep 23 17:39:29 dist-test-slave-ppjl krb5kdc[6135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:39:29.317550 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:29.299625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:33837 (local address 127.20.240.126:39185)
0923 17:39:29.299857 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:29.299864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:29.301057 (+ 1193us) server_negotiation.cc:407] Connection header received
0923 17:39:29.302968 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:29.302973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:29.303047 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:29.303139 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:29.304706 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.305426 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.307028 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.307601 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.307718 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:29.311689 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:29.311694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:29.311697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:29.311730 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:29.314313 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:29.315242 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:29.315246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:29.315248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:29.315390 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:29.315845 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:29.315848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:29.315850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:29.316138 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:29.316265 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:29.317099 (+ 834us) server_negotiation.cc:299] Negotiation successful
0923 17:39:29.317288 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260923 17:39:29.318984 6505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:29.300155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:39185 (local address 127.20.240.66:33837)
0923 17:39:29.300907 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:29.300952 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:29.302707 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:29.303416 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:29.303425 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:29.303946 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:29.304533 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:29.304552 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.305556 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.305560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:29.306889 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:29.306901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.307819 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.307826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:29.308081 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:29.308770 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:39:29.308802 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:29.311436 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:29.314460 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:29.314470 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:29.314490 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:29.315069 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:29.315495 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:29.315500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:29.315503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:29.315746 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:29.316258 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:29.316266 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:29.316695 (+ 429us) client_negotiation.cc:769] Sending connection context
0923 17:39:29.316981 (+ 286us) client_negotiation.cc:240] Negotiation successful
0923 17:39:29.317545 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":283,"threads_started":1}
I20260923 17:39:29.320511 6503 heartbeater.cc:347] Connected to a master server at 127.20.240.126:39185
I20260923 17:39:29.320858 6503 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:29.321705 6503 heartbeater.cc:511] Master 127.20.240.126:39185 requested a full tablet report, sending...
I20260923 17:39:29.323351 6180 ts_manager.cc:194] Registered new tserver with Master: a145f8d4584445e0935e565fed5651f1 (127.20.240.66:45431)
I20260923 17:39:29.324184 6180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:33837
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:39:29.363075 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:39185
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:29.707248 6508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:29.707599 6508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:29.707741 6508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:29.715409 6508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:29.715549 6508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:29.715700 6508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:39:29.725924 6508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:29.727283 6508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:29.728652 6508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:29.741837 6516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:29.741812 6514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:29.741776 6513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:29.742981 6508 server_base.cc:1034] running on GCE node
I20260923 17:39:29.743623 6508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:29.744630 6508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:29.745839 6508 hybrid_clock.cc:648] HybridClock initialized: now 1790185169745782 us; error 42 us; skew 500 ppm
I20260923 17:39:29.746205 6508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:29 dist-test-slave-ppjl krb5kdc[6135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:29.749370 6508 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:39:29.750837 6508 webserver.cc:460] Webserver started at http://127.20.240.67:40255/ using document root <none> and password file <none>
I20260923 17:39:29.751580 6508 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:29.751690 6508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:29.751962 6508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:29.754908 6508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "581572a73c30416e9ab006303c031863"
format_stamp: "Formatted at 2026-09-23 17:39:29 on dist-test-slave-ppjl"
server_key: "3dd08eb50c4306c5b34da257df1c0157"
server_key_iv: "b95e569d1212f122e8d1492b759aee58"
server_key_version: "encryptionkey@0"
I20260923 17:39:29.755718 6508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "581572a73c30416e9ab006303c031863"
format_stamp: "Formatted at 2026-09-23 17:39:29 on dist-test-slave-ppjl"
server_key: "3dd08eb50c4306c5b34da257df1c0157"
server_key_iv: "b95e569d1212f122e8d1492b759aee58"
server_key_version: "encryptionkey@0"
I20260923 17:39:29.761139 6508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:39:29.764858 6523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:29.766359 6508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:39:29.766532 6508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "581572a73c30416e9ab006303c031863"
format_stamp: "Formatted at 2026-09-23 17:39:29 on dist-test-slave-ppjl"
server_key: "3dd08eb50c4306c5b34da257df1c0157"
server_key_iv: "b95e569d1212f122e8d1492b759aee58"
server_key_version: "encryptionkey@0"
I20260923 17:39:29.766731 6508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:29.789827 6508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:29.794203 6508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:29.794471 6508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:29.795524 6508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:29.796976 6508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:29.797056 6508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:29.797155 6508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:29.797199 6508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:29.813313 6369 heartbeater.cc:503] Master 127.20.240.126:39185 was elected leader, sending a full tablet report...
I20260923 17:39:29.830420 6508 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:33741
I20260923 17:39:29.830503 6635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:33741 every 8 connection(s)
I20260923 17:39:29.831773 6508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:39:29.841599 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 6508
I20260923 17:39:29.841792 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:39:29.842239 21441 external_mini_cluster.cc:1428] Setting key 17faa49f26692cef9967887df5362b7d
Sep 23 17:39:29 dist-test-slave-ppjl krb5kdc[6135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185169, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:39:29.851424 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:29.834342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:36861 (local address 127.20.240.126:39185)
0923 17:39:29.834801 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:29.834805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:29.835721 (+ 916us) server_negotiation.cc:407] Connection header received
0923 17:39:29.837676 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:29.837680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:29.837761 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:29.837856 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:29.839374 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.840170 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.841991 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.842433 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.842529 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:29.846076 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:29.846081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:29.846084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:29.846123 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:29.848444 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:29.849204 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:29.849209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:29.849211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:29.849347 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:29.849765 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:29.849770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:29.849772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:29.850113 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:29.850244 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:29.850921 (+ 677us) server_negotiation.cc:299] Negotiation successful
0923 17:39:29.851104 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":223,"threads_started":1}
I20260923 17:39:29.852368 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:29.834823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:39185 (local address 127.20.240.67:36861)
0923 17:39:29.835552 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:29.835598 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:29.837374 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:29.838108 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:29.838117 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:29.838616 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:29.839215 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:29.839233 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.840302 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.840305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:29.841803 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:29.841819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.842650 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.842658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:29.842885 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:29.843530 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:39:29.843554 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:29.845894 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:29.848568 (+ 2674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:29.848574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:29.848588 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:29.849012 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:29.849432 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:29.849435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:29.849438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:29.849621 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:29.850236 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:29.850243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:29.850572 (+ 329us) client_negotiation.cc:769] Sending connection context
0923 17:39:29.850791 (+ 219us) client_negotiation.cc:240] Negotiation successful
0923 17:39:29.851239 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":295,"threads_started":1}
I20260923 17:39:29.853837 6636 heartbeater.cc:347] Connected to a master server at 127.20.240.126:39185
I20260923 17:39:29.854120 6636 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:29.854776 6636 heartbeater.cc:511] Master 127.20.240.126:39185 requested a full tablet report, sending...
I20260923 17:39:29.856398 6180 ts_manager.cc:194] Registered new tserver with Master: 581572a73c30416e9ab006303c031863 (127.20.240.67:33741)
I20260923 17:39:29.857338 6180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:36861
I20260923 17:39:29.857671 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:39:29.875602 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:29.862887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.20.240.126:39185)
0923 17:39:29.863098 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:29.863104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:29.863492 (+ 388us) server_negotiation.cc:407] Connection header received
0923 17:39:29.863652 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:29.863656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:29.863720 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:29.863802 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:29.865271 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.865996 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.867428 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.867864 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.867958 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:29.870496 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:29.870501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:29.870504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:29.870539 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:29.873192 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:29.873903 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:29.873907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:29.873909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:29.874021 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:29.874317 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:29.874320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:29.874322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:29.874651 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:29.874819 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:29.875079 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 17:39:29.875298 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:39:29.879724 6180 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:39:29.886754 6180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:39:29.919727 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:29.901135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.20.240.67:33741)
0923 17:39:29.910597 (+ 9462us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:29.910607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:29.910642 (+ 35us) server_negotiation.cc:407] Connection header received
0923 17:39:29.910751 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:29.910759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:29.911061 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:29.912747 (+ 1686us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:29.913050 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.914675 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.916987 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.918288 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.918418 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:29.918697 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:29.919222 (+ 525us) server_negotiation.cc:299] Negotiation successful
0923 17:39:29.919390 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9271,"thread_start_us":9114,"threads_started":1}
I20260923 17:39:29.919780 6651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:29.900902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:33741 (local address 127.0.0.1:47860)
0923 17:39:29.909611 (+ 8709us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:29.909686 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:29.909885 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:29.912090 (+ 2205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:29.912098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:29.912136 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:29.912539 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:29.912554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.914826 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.914832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:29.916808 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:29.916822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.918830 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.918836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:29.918967 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:29.919003 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:39:29.919133 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 17:39:29.919375 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8411,"mutex_wait_us":70,"thread_start_us":10500,"threads_started":1}
I20260923 17:39:29.920697 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:29.910513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.20.240.65:43963)
0923 17:39:29.911185 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:29.911194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:29.912010 (+ 816us) server_negotiation.cc:407] Connection header received
0923 17:39:29.912149 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:29.912156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:29.912366 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:29.912552 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:29.913673 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.914686 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.917885 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.919397 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.919506 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:29.919720 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:29.920309 (+ 589us) server_negotiation.cc:299] Negotiation successful
0923 17:39:29.920466 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":333,"threads_started":1}
I20260923 17:39:29.920732 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:29.911203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:43963 (local address 127.0.0.1:56982)
0923 17:39:29.911872 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:29.911895 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:29.912014 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:29.912823 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:29.912827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:29.913000 (+ 173us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:29.913514 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:29.913527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.915267 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.915272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:29.917692 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:29.917707 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.919909 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.919916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:29.920089 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:29.920122 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:39:29.920210 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 17:39:29.920331 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":72,"thread_start_us":361,"threads_started":1}
I20260923 17:39:29.921805 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:29.901296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:45431 (local address 127.0.0.1:32824)
0923 17:39:29.911723 (+ 10427us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:29.911745 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:29.911864 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:29.912727 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:29.912732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:29.912756 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:29.913259 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:29.913274 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.915101 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.915104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:29.917698 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:29.917707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.921016 (+ 3309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.921025 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:29.921193 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:29.921223 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 17:39:29.921318 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 17:39:29.921446 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10292,"mutex_wait_us":62,"thread_start_us":7870,"threads_started":1}
I20260923 17:39:29.921850 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:29.901820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32824 (local address 127.20.240.66:45431)
0923 17:39:29.907734 (+ 5914us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:29.907746 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:29.911861 (+ 4115us) server_negotiation.cc:407] Connection header received
0923 17:39:29.912001 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:29.912009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:29.912267 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:29.912470 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:29.913544 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.914961 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.917884 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:29.919397 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:29.919511 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:29.919690 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:29.921370 (+ 1680us) server_negotiation.cc:299] Negotiation successful
0923 17:39:29.921557 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5710,"thread_start_us":5561,"threads_started":1}
I20260923 17:39:29.924445 6304 tablet_service.cc:1467] Processing CreateTablet for tablet cb2c3a1cb56a4e6da0625dfb0f4a8e3e (DEFAULT_TABLE table=test-table [id=b825ab0e9fd243aeb43a21676d8b81ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:29.924445 6571 tablet_service.cc:1467] Processing CreateTablet for tablet cb2c3a1cb56a4e6da0625dfb0f4a8e3e (DEFAULT_TABLE table=test-table [id=b825ab0e9fd243aeb43a21676d8b81ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:29.924820 6438 tablet_service.cc:1467] Processing CreateTablet for tablet cb2c3a1cb56a4e6da0625dfb0f4a8e3e (DEFAULT_TABLE table=test-table [id=b825ab0e9fd243aeb43a21676d8b81ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:29.925865 6438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2c3a1cb56a4e6da0625dfb0f4a8e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:29.925938 6571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2c3a1cb56a4e6da0625dfb0f4a8e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:29.925938 6304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb2c3a1cb56a4e6da0625dfb0f4a8e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:29.937422 6655 tablet_bootstrap.cc:492] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1: Bootstrap starting.
I20260923 17:39:29.940718 6656 tablet_bootstrap.cc:492] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb: Bootstrap starting.
I20260923 17:39:29.940969 6655 tablet_bootstrap.cc:654] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:29.942601 6655 log.cc:824] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:29.945012 6656 tablet_bootstrap.cc:654] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:29.945739 6657 tablet_bootstrap.cc:492] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863: Bootstrap starting.
I20260923 17:39:29.945924 6655 tablet_bootstrap.cc:492] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1: No bootstrap required, opened a new log
I20260923 17:39:29.946202 6655 ts_tablet_manager.cc:1397] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260923 17:39:29.946344 6656 log.cc:824] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:29.949198 6657 tablet_bootstrap.cc:654] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:29.949725 6656 tablet_bootstrap.cc:492] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb: No bootstrap required, opened a new log
I20260923 17:39:29.950030 6656 ts_tablet_manager.cc:1397] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 17:39:29.950518 6657 log.cc:824] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:29.953966 6657 tablet_bootstrap.cc:492] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863: No bootstrap required, opened a new log
I20260923 17:39:29.954258 6657 ts_tablet_manager.cc:1397] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260923 17:39:29.954001 6655 raft_consensus.cc:348] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a145f8d4584445e0935e565fed5651f1" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45431 } } peers { permanent_uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 43963 } } peers { permanent_uuid: "581572a73c30416e9ab006303c031863" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 33741 } }
I20260923 17:39:29.954521 6655 raft_consensus.cc:374] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:29.954625 6655 raft_consensus.cc:729] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a145f8d4584445e0935e565fed5651f1, State: Initialized, Role: FOLLOWER
I20260923 17:39:29.955315 6655 consensus_queue.cc:260] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a145f8d4584445e0935e565fed5651f1" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45431 } } peers { permanent_uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 43963 } } peers { permanent_uuid: "581572a73c30416e9ab006303c031863" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 33741 } }
I20260923 17:39:29.957072 6503 heartbeater.cc:503] Master 127.20.240.126:39185 was elected leader, sending a full tablet report...
I20260923 17:39:29.957808 6655 ts_tablet_manager.cc:1428] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260923 17:39:29.960242 6656 raft_consensus.cc:348] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a145f8d4584445e0935e565fed5651f1" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45431 } } peers { permanent_uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 43963 } } peers { permanent_uuid: "581572a73c30416e9ab006303c031863" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 33741 } }
I20260923 17:39:29.960714 6656 raft_consensus.cc:374] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:29.960820 6656 raft_consensus.cc:729] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b57b3efbbde4d5ebb20d5ccbb4b86fb, State: Initialized, Role: FOLLOWER
I20260923 17:39:29.961531 6656 consensus_queue.cc:260] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a145f8d4584445e0935e565fed5651f1" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45431 } } peers { permanent_uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 43963 } } peers { permanent_uuid: "581572a73c30416e9ab006303c031863" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 33741 } }
I20260923 17:39:29.963434 6656 ts_tablet_manager.cc:1428] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260923 17:39:29.964874 6657 raft_consensus.cc:348] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a145f8d4584445e0935e565fed5651f1" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45431 } } peers { permanent_uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 43963 } } peers { permanent_uuid: "581572a73c30416e9ab006303c031863" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 33741 } }
I20260923 17:39:29.965216 6657 raft_consensus.cc:374] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:29.965286 6657 raft_consensus.cc:729] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 581572a73c30416e9ab006303c031863, State: Initialized, Role: FOLLOWER
I20260923 17:39:29.965817 6657 consensus_queue.cc:260] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a145f8d4584445e0935e565fed5651f1" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45431 } } peers { permanent_uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 43963 } } peers { permanent_uuid: "581572a73c30416e9ab006303c031863" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 33741 } }
I20260923 17:39:29.967036 6636 heartbeater.cc:503] Master 127.20.240.126:39185 was elected leader, sending a full tablet report...
I20260923 17:39:29.967514 6657 ts_tablet_manager.cc:1428] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260923 17:39:30.004917 6661 raft_consensus.cc:482] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:39:30.005275 6661 raft_consensus.cc:504] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a145f8d4584445e0935e565fed5651f1" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45431 } } peers { permanent_uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 43963 } } peers { permanent_uuid: "581572a73c30416e9ab006303c031863" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 33741 } }
I20260923 17:39:30.006935 6661 leader_election.cc:288] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b57b3efbbde4d5ebb20d5ccbb4b86fb (127.20.240.65:43963), 581572a73c30416e9ab006303c031863 (127.20.240.67:33741)
I20260923 17:39:30.013558 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.007441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:33719 (local address 127.20.240.67:33741)
0923 17:39:30.007762 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:30.007769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:30.008429 (+ 660us) server_negotiation.cc:407] Connection header received
0923 17:39:30.008583 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:30.008587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:30.008667 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:30.008743 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:30.009335 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.010056 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.011853 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.012672 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.012830 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:30.012907 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:30.013200 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 17:39:30.013324 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260923 17:39:30.013700 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.007327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:53373 (local address 127.20.240.65:43963)
0923 17:39:30.007548 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:30.007554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:30.008318 (+ 764us) server_negotiation.cc:407] Connection header received
0923 17:39:30.008515 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:30.008518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:30.008582 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:30.008658 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:30.009205 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.009909 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.011885 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.012782 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.012928 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:30.013004 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:30.013321 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 17:39:30.013445 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:39:30.014002 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.007463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:33741 (local address 127.20.240.66:33719)
0923 17:39:30.008319 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:30.008333 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:30.008456 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:30.008773 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:30.008778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:30.008817 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:30.009186 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:30.009194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.010200 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.010203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:30.011697 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:30.011713 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.012883 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.012890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:30.013044 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:30.013068 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:39:30.013155 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 17:39:30.013286 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":2,"thread_start_us":505,"threads_started":1}
I20260923 17:39:30.014061 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.007365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:43963 (local address 127.20.240.66:53373)
0923 17:39:30.008195 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:30.008216 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:30.008370 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:30.008685 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:30.008690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:30.008718 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:30.009057 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:30.009067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.010040 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.010045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:30.011736 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:30.011749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.013003 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.013010 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:30.013155 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:30.013186 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 17:39:30.013267 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:39:30.013446 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":73,"thread_start_us":442,"threads_started":1}
I20260923 17:39:30.014928 6324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2c3a1cb56a4e6da0625dfb0f4a8e3e" candidate_uuid: "a145f8d4584445e0935e565fed5651f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb" is_pre_election: true
I20260923 17:39:30.014930 6591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2c3a1cb56a4e6da0625dfb0f4a8e3e" candidate_uuid: "a145f8d4584445e0935e565fed5651f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "581572a73c30416e9ab006303c031863" is_pre_election: true
I20260923 17:39:30.015429 6591 raft_consensus.cc:2454] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a145f8d4584445e0935e565fed5651f1 in term 0.
I20260923 17:39:30.015444 6324 raft_consensus.cc:2454] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a145f8d4584445e0935e565fed5651f1 in term 0.
I20260923 17:39:30.016068 6392 leader_election.cc:302] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b57b3efbbde4d5ebb20d5ccbb4b86fb, a145f8d4584445e0935e565fed5651f1; no voters:
I20260923 17:39:30.016460 6661 raft_consensus.cc:2781] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:39:30.016614 6661 raft_consensus.cc:482] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:39:30.016716 6661 raft_consensus.cc:3037] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:30.018888 6661 raft_consensus.cc:504] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a145f8d4584445e0935e565fed5651f1" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45431 } } peers { permanent_uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 43963 } } peers { permanent_uuid: "581572a73c30416e9ab006303c031863" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 33741 } }
I20260923 17:39:30.019574 6661 leader_election.cc:288] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [CANDIDATE]: Term 1 election: Requested vote from peers 0b57b3efbbde4d5ebb20d5ccbb4b86fb (127.20.240.65:43963), 581572a73c30416e9ab006303c031863 (127.20.240.67:33741)
I20260923 17:39:30.020031 6324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2c3a1cb56a4e6da0625dfb0f4a8e3e" candidate_uuid: "a145f8d4584445e0935e565fed5651f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb"
I20260923 17:39:30.020097 6591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb2c3a1cb56a4e6da0625dfb0f4a8e3e" candidate_uuid: "a145f8d4584445e0935e565fed5651f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "581572a73c30416e9ab006303c031863"
I20260923 17:39:30.020311 6324 raft_consensus.cc:3037] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:30.020321 6591 raft_consensus.cc:3037] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:30.022572 6591 raft_consensus.cc:2454] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a145f8d4584445e0935e565fed5651f1 in term 1.
I20260923 17:39:30.022585 6324 raft_consensus.cc:2454] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a145f8d4584445e0935e565fed5651f1 in term 1.
I20260923 17:39:30.023046 6393 leader_election.cc:302] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 581572a73c30416e9ab006303c031863, a145f8d4584445e0935e565fed5651f1; no voters:
I20260923 17:39:30.023461 6661 raft_consensus.cc:2781] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:30.023888 6661 raft_consensus.cc:686] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [term 1 LEADER]: Becoming Leader. State: Replica: a145f8d4584445e0935e565fed5651f1, State: Running, Role: LEADER
I20260923 17:39:30.024420 6661 consensus_queue.cc:237] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a145f8d4584445e0935e565fed5651f1" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45431 } } peers { permanent_uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 43963 } } peers { permanent_uuid: "581572a73c30416e9ab006303c031863" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 33741 } }
I20260923 17:39:30.029722 6180 catalog_manager.cc:5495] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a145f8d4584445e0935e565fed5651f1 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "a145f8d4584445e0935e565fed5651f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a145f8d4584445e0935e565fed5651f1" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 43963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "581572a73c30416e9ab006303c031863" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 33741 } health_report { overall_health: UNKNOWN } } }
W20260923 17:39:30.030929 6370 tablet.cc:2369] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:39:30.049207 6504 tablet.cc:2369] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:39:30.073460 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.067568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.20.240.66:45431)
0923 17:39:30.067792 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:30.067798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:30.067840 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:39:30.068149 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:30.068154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:30.068249 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:30.068381 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:39:30.068771 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.069484 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.071085 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.071582 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.071694 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:30.072275 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:39:30.072824 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:39:30.072963 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:30.073051 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 17:39:30.073184 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:39:30.082198 6591 raft_consensus.cc:1261] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863 [term 1 FOLLOWER]: Refusing update from remote peer a145f8d4584445e0935e565fed5651f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:39:30.082201 6324 raft_consensus.cc:1261] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb [term 1 FOLLOWER]: Refusing update from remote peer a145f8d4584445e0935e565fed5651f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260923 17:39:30.083438 6637 tablet.cc:2369] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:39:30.083259 6666 consensus_queue.cc:1035] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b57b3efbbde4d5ebb20d5ccbb4b86fb" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 43963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:39:30.083693 6661 consensus_queue.cc:1035] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "581572a73c30416e9ab006303c031863" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 33741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:39:30.095948 6671 mvcc.cc:205] Tried to move back new op lower bound from 7332598456646144000 to 7332598456424161280. Current Snapshot: MvccSnapshot[applied={T|T < 7332598456646144000}]
I20260923 17:39:30.109195 6180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:39:30.109452 6180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:39:30.113543 6180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34316b9d0110472ea6a380a0fe1b7513: Sending DeleteTablet for 3 replicas of tablet cb2c3a1cb56a4e6da0625dfb0f4a8e3e
I20260923 17:39:30.115000 6438 tablet_service.cc:1514] Processing DeleteTablet for tablet cb2c3a1cb56a4e6da0625dfb0f4a8e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:30 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:32824
I20260923 17:39:30.115039 6304 tablet_service.cc:1514] Processing DeleteTablet for tablet cb2c3a1cb56a4e6da0625dfb0f4a8e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:30 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:56982
I20260923 17:39:30.115098 6571 tablet_service.cc:1514] Processing DeleteTablet for tablet cb2c3a1cb56a4e6da0625dfb0f4a8e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:30 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:47860
I20260923 17:39:30.118762 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 6241
I20260923 17:39:30.127436 6679 tablet_replica.cc:330] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1: stopping tablet replica
I20260923 17:39:30.128188 6679 raft_consensus.cc:2229] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:39:30.129076 6679 raft_consensus.cc:2258] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:30.133235 6679 ts_tablet_manager.cc:1905] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:30.139016 6681 tablet_replica.cc:330] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb: stopping tablet replica
I20260923 17:39:30.139582 6681 raft_consensus.cc:2229] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:39:30.140178 6681 raft_consensus.cc:2258] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:30.141140 6680 tablet_replica.cc:330] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863: stopping tablet replica
Sep 23 17:39:30 dist-test-slave-ppjl krb5kdc[6135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:39:30.148993 6679 ts_tablet_manager.cc:1918] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:30.149241 6679 log.cc:1196] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/cb2c3a1cb56a4e6da0625dfb0f4a8e3e
I20260923 17:39:30.149801 6679 ts_tablet_manager.cc:1939] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P a145f8d4584445e0935e565fed5651f1: Deleting consensus metadata
I20260923 17:39:30.151867 6165 catalog_manager.cc:4841] TS a145f8d4584445e0935e565fed5651f1 (127.20.240.66:45431): tablet cb2c3a1cb56a4e6da0625dfb0f4a8e3e (table test-table [id=b825ab0e9fd243aeb43a21676d8b81ee]) successfully deleted
I20260923 17:39:30.153039 6681 ts_tablet_manager.cc:1905] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 0b57b3efbbde4d5ebb20d5ccbb4b86fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:30.153987 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.122329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56986 (local address 127.20.240.65:43963)
0923 17:39:30.122540 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:30.122547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:30.127732 (+ 5185us) server_negotiation.cc:407] Connection header received
0923 17:39:30.128127 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:30.128134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:30.128246 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:30.128349 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:30.130413 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.131639 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.133735 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.134465 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.134583 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:30.143100 (+ 8517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:30.143121 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:30.143141 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:30.143219 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:30.149709 (+ 6490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:30.150902 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:30.150913 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:30.150917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:30.151120 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:30.151648 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:30.151653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:30.151655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:30.152156 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:30.152539 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:30.153369 (+ 830us) server_negotiation.cc:299] Negotiation successful
0923 17:39:30.153611 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:39:30.141602 6680 raft_consensus.cc:2229] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:39:30.154709 6680 raft_consensus.cc:2258] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:30.155530 6364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:30.157893 6680 ts_tablet_manager.cc:1905] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:30.166319 6680 ts_tablet_manager.cc:1918] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:30.166669 6680 log.cc:1196] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/cb2c3a1cb56a4e6da0625dfb0f4a8e3e
I20260923 17:39:30.167316 6680 ts_tablet_manager.cc:1939] T cb2c3a1cb56a4e6da0625dfb0f4a8e3e P 581572a73c30416e9ab006303c031863: Deleting consensus metadata
I20260923 17:39:30.169639 6166 catalog_manager.cc:4841] TS 581572a73c30416e9ab006303c031863 (127.20.240.67:33741): tablet cb2c3a1cb56a4e6da0625dfb0f4a8e3e (table test-table [id=b825ab0e9fd243aeb43a21676d8b81ee]) successfully deleted
I20260923 17:39:30.258862 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 6241
W20260923 17:39:30.279886 6165 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.240.65:43963 (error 108)
I20260923 17:39:30.280035 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 6375
I20260923 17:39:30.282636 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.282139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:43963 (local address 127.0.0.1:57002)
0923 17:39:30.282325 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:30.282452 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260923 17:39:30.282939 6165 catalog_manager.cc:4584] TS 0b57b3efbbde4d5ebb20d5ccbb4b86fb (127.20.240.65:43963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb2c3a1cb56a4e6da0625dfb0f4a8e3e: Network error: Client connection negotiation failed: client connection to 127.20.240.65:43963: connect: Connection refused (error 111)
Sep 23 17:39:30 dist-test-slave-ppjl krb5kdc[6135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:39:30.298863 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.280846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.20.240.66:45431)
0923 17:39:30.281041 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:30.281047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:30.281141 (+ 94us) server_negotiation.cc:407] Connection header received
0923 17:39:30.281792 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:30.281798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:30.281885 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:30.281987 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:30.283742 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.284976 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.286310 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.286993 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.287135 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:30.291537 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:30.291558 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:30.291580 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:30.291640 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:30.295555 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:30.296446 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:30.296455 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:30.296462 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:30.296636 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:30.297074 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:30.297083 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:30.297087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:30.297579 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:30.297936 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:30.298235 (+ 299us) server_negotiation.cc:299] Negotiation successful
0923 17:39:30.298467 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:39:30.299695 6498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:30.336982 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.336396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:43963 (local address 127.0.0.1:57006)
0923 17:39:30.336642 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:30.336769 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 17:39:30.338038 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.337680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:43963 (local address 127.0.0.1:57022)
0923 17:39:30.337829 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:30.337921 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 17:39:30.398500 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.398006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:43963 (local address 127.0.0.1:57038)
0923 17:39:30.398236 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:30.398332 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260923 17:39:30.399586 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.399233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:43963 (local address 127.0.0.1:57048)
0923 17:39:30.399378 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:30.399456 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260923 17:39:30.413497 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 6375
I20260923 17:39:30.429183 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 6508
Sep 23 17:39:30 dist-test-slave-ppjl krb5kdc[6135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:39:30.445962 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.429926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.20.240.67:33741)
0923 17:39:30.430098 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:30.430104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:30.430180 (+ 76us) server_negotiation.cc:407] Connection header received
0923 17:39:30.430373 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:30.430377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:30.430442 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:30.430530 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:30.431914 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.432659 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.434126 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:30.434533 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:30.434660 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:30.439518 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:30.439536 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:30.439560 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:30.439608 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:30.442492 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:30.443370 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:30.443379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:30.443381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:30.443637 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:30.444220 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:30.444224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:30.444226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:30.444689 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:30.445003 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:30.445361 (+ 358us) server_negotiation.cc:299] Negotiation successful
0923 17:39:30.445621 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:39:30.446705 6631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:30.480450 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.479923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:43963 (local address 127.0.0.1:57052)
0923 17:39:30.480187 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:30.480291 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260923 17:39:30.481673 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.481237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:43963 (local address 127.0.0.1:57054)
0923 17:39:30.481470 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:30.481548 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260923 17:39:30.537365 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 6508
I20260923 17:39:30.550493 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 6148
I20260923 17:39:30.551822 6210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:30.663244 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 6148
I20260923 17:39:30.663311 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:30.662692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:43963 (local address 127.0.0.1:57060)
0923 17:39:30.662960 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:30.663068 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:43963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
2026-09-23T17:39:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4232 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T17:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:39:30Z Disabled control of system clock
I20260923 17:39:30.739522 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41547
--webserver_interface=127.20.240.126
--webserver_port=0
--builtin_ntp_servers=127.20.240.84:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:41547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:39:31.076176 6691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:31.076465 6691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:31.076546 6691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:31.083904 6691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:39:31.084028 6691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:31.084072 6691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:31.084121 6691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:39:31.084165 6691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:39:31.093703 6691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:41547
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:31.094967 6691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:31.096767 6691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:31.108568 6697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:31.108568 6696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:31.108609 6699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:31.109380 6691 server_base.cc:1034] running on GCE node
I20260923 17:39:31.110205 6691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:31.111719 6691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:31.112936 6691 hybrid_clock.cc:648] HybridClock initialized: now 1790185171112885 us; error 36 us; skew 500 ppm
I20260923 17:39:31.113302 6691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:39:31.121984 6691 webserver.cc:460] Webserver started at http://127.20.240.126:44025/ using document root <none> and password file <none>
I20260923 17:39:31.122869 6691 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:31.122995 6691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:31.123355 6691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:31.126403 6691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "6b828f366d4e414da2b4ea006f6bd00b"
format_stamp: "Formatted at 2026-09-23 17:39:31 on dist-test-slave-ppjl"
server_key: "0063e3768f786645e802dfa3ab639669"
server_key_iv: "631fad85ca4391cedb224d677b06374f"
server_key_version: "encryptionkey@0"
I20260923 17:39:31.127255 6691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "6b828f366d4e414da2b4ea006f6bd00b"
format_stamp: "Formatted at 2026-09-23 17:39:31 on dist-test-slave-ppjl"
server_key: "0063e3768f786645e802dfa3ab639669"
server_key_iv: "631fad85ca4391cedb224d677b06374f"
server_key_version: "encryptionkey@0"
I20260923 17:39:31.132495 6691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 17:39:31.136173 6705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:31.137604 6691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260923 17:39:31.137777 6691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "6b828f366d4e414da2b4ea006f6bd00b"
format_stamp: "Formatted at 2026-09-23 17:39:31 on dist-test-slave-ppjl"
server_key: "0063e3768f786645e802dfa3ab639669"
server_key_iv: "631fad85ca4391cedb224d677b06374f"
server_key_version: "encryptionkey@0"
I20260923 17:39:31.137984 6691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:31.156275 6691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:31.157158 6691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:31.157436 6691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:31.175405 6691 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:41547
I20260923 17:39:31.175418 6756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:41547 every 8 connection(s)
I20260923 17:39:31.176801 6691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:39:31.181010 6757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:31.183501 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 6691
I20260923 17:39:31.183698 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:39:31.184258 21441 external_mini_cluster.cc:1428] Setting key 2a49c95ca5524c6fc228f5898149bc43
I20260923 17:39:31.193449 6757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b: Bootstrap starting.
I20260923 17:39:31.196656 6757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:31.197768 6757 log.cc:824] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:31.198032 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:31.186280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.20.240.126:41547)
0923 17:39:31.187364 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:31.187375 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:31.187422 (+ 47us) server_negotiation.cc:407] Connection header received
0923 17:39:31.189135 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:31.189160 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:31.189645 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:31.190001 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:31.190619 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:31.192043 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:31.193523 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:31.194254 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:31.194434 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:31.195590 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:31.195598 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:31.195614 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:31.196158 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:31.196250 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:31.196277 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:31.196377 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:31.196622 (+ 245us) server_negotiation.cc:299] Negotiation successful
0923 17:39:31.196970 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":339,"threads_started":1}
I20260923 17:39:31.201143 6757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b: No bootstrap required, opened a new log
I20260923 17:39:31.208905 6757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b828f366d4e414da2b4ea006f6bd00b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41547 } }
I20260923 17:39:31.209286 6757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:31.209367 6757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b828f366d4e414da2b4ea006f6bd00b, State: Initialized, Role: FOLLOWER
I20260923 17:39:31.209870 6757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b828f366d4e414da2b4ea006f6bd00b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41547 } }
I20260923 17:39:31.210079 6757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:39:31.210173 6757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:39:31.210289 6757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:31.212702 6757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b828f366d4e414da2b4ea006f6bd00b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41547 } }
I20260923 17:39:31.213286 6757 leader_election.cc:302] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b828f366d4e414da2b4ea006f6bd00b; no voters:
I20260923 17:39:31.213840 6757 leader_election.cc:288] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:39:31.214015 6762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:31.214666 6762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [term 1 LEADER]: Becoming Leader. State: Replica: 6b828f366d4e414da2b4ea006f6bd00b, State: Running, Role: LEADER
I20260923 17:39:31.215195 6762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b828f366d4e414da2b4ea006f6bd00b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41547 } }
I20260923 17:39:31.215898 6757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:39:31.217955 6764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b828f366d4e414da2b4ea006f6bd00b. Latest consensus state: current_term: 1 leader_uuid: "6b828f366d4e414da2b4ea006f6bd00b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b828f366d4e414da2b4ea006f6bd00b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41547 } } }
I20260923 17:39:31.218369 6764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:31.217952 6763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b828f366d4e414da2b4ea006f6bd00b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b828f366d4e414da2b4ea006f6bd00b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41547 } } }
I20260923 17:39:31.218662 6763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:31.220723 6768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:39:31.226243 6768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:39:31.235126 6768 catalog_manager.cc:1347] Generated new cluster ID: 2dedf0744568483fb0e51726884bc5f5
I20260923 17:39:31.235284 6768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:39:31.257202 6768 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:39:31.259214 6768 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:39:31.266445 6768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b: Generated new TSK 0
I20260923 17:39:31.267541 6768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:39:31.285157 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41547
--builtin_ntp_servers=127.20.240.84:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:39:31.613974 6781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:31.614264 6781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:31.614346 6781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:31.621654 6781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:31.621760 6781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:31.621919 6781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:39:31.631477 6781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:31.632676 6781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:31.634447 6781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:31.647711 6786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:31.647684 6787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:31.647747 6789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:31.648783 6781 server_base.cc:1034] running on GCE node
I20260923 17:39:31.649499 6781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:31.650462 6781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:31.651680 6781 hybrid_clock.cc:648] HybridClock initialized: now 1790185171651645 us; error 58 us; skew 500 ppm
I20260923 17:39:31.652034 6781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:39:31.654353 6781 webserver.cc:460] Webserver started at http://127.20.240.65:41425/ using document root <none> and password file <none>
I20260923 17:39:31.655094 6781 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:31.655228 6781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:31.655526 6781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:31.658664 6781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "fc978b8f03bd43c7a5a16845044b104d"
format_stamp: "Formatted at 2026-09-23 17:39:31 on dist-test-slave-ppjl"
server_key: "aaba1658ac9617d84e52936dcac849cb"
server_key_iv: "05bb0d183ceac191517166b04a5ecc9f"
server_key_version: "encryptionkey@0"
I20260923 17:39:31.659579 6781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "fc978b8f03bd43c7a5a16845044b104d"
format_stamp: "Formatted at 2026-09-23 17:39:31 on dist-test-slave-ppjl"
server_key: "aaba1658ac9617d84e52936dcac849cb"
server_key_iv: "05bb0d183ceac191517166b04a5ecc9f"
server_key_version: "encryptionkey@0"
I20260923 17:39:31.665316 6781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:39:31.669241 6795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:31.670940 6781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:39:31.671191 6781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "fc978b8f03bd43c7a5a16845044b104d"
format_stamp: "Formatted at 2026-09-23 17:39:31 on dist-test-slave-ppjl"
server_key: "aaba1658ac9617d84e52936dcac849cb"
server_key_iv: "05bb0d183ceac191517166b04a5ecc9f"
server_key_version: "encryptionkey@0"
I20260923 17:39:31.671432 6781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:31.689844 6781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:31.690714 6781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:31.691015 6781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:31.692092 6781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:31.693533 6781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:31.693629 6781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:31.693733 6781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:31.693804 6781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:31.725046 6781 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:40181
I20260923 17:39:31.725083 6907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:40181 every 8 connection(s)
I20260923 17:39:31.726411 6781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:39:31.728704 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 6781
I20260923 17:39:31.728873 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:39:31.729385 21441 external_mini_cluster.cc:1428] Setting key 80903c7286bc3df26478b947e0e263e1
I20260923 17:39:31.732093 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41547
--builtin_ntp_servers=127.20.240.84:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:39:31.739032 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:31.729057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:41455 (local address 127.20.240.126:41547)
0923 17:39:31.729549 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:31.729555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:31.731228 (+ 1673us) server_negotiation.cc:407] Connection header received
0923 17:39:31.732775 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:31.732780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:31.732863 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:31.732955 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:31.734064 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:31.734767 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:31.736324 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:31.736732 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:31.736842 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:31.738177 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:31.738182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:31.738185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:31.738205 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:31.738246 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:31.738258 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:31.738331 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:31.738545 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 17:39:31.738667 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":227,"threads_started":1}
I20260923 17:39:31.740074 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:31.729690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41547 (local address 127.20.240.65:41455)
0923 17:39:31.730771 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:31.730824 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:31.732520 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:31.733203 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:31.733212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:31.733306 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:31.733888 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:31.733905 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:31.734905 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:31.734909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:31.736185 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:31.736197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:31.736951 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:31.736957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:31.737190 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:31.737854 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:39:31.737856 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:31.737899 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:39:31.737990 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:31.738345 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:31.738356 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:31.738365 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:39:31.738533 (+ 168us) client_negotiation.cc:240] Negotiation successful
0923 17:39:31.738984 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":345,"threads_started":1}
I20260923 17:39:31.741818 6908 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41547
I20260923 17:39:31.742158 6908 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:31.743072 6908 heartbeater.cc:511] Master 127.20.240.126:41547 requested a full tablet report, sending...
I20260923 17:39:31.745503 6722 ts_manager.cc:194] Registered new tserver with Master: fc978b8f03bd43c7a5a16845044b104d (127.20.240.65:40181)
I20260923 17:39:31.747627 6722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.65:41455
W20260923 17:39:32.063714 6912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:32.064034 6912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:32.064127 6912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:32.072119 6912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:32.072240 6912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:32.072350 6912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:39:32.081869 6912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:32.083184 6912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:32.084949 6912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:32.097941 6918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:32.097961 6920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:32.098013 6917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:32.098549 6912 server_base.cc:1034] running on GCE node
I20260923 17:39:32.099181 6912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:32.100140 6912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:32.101339 6912 hybrid_clock.cc:648] HybridClock initialized: now 1790185172101290 us; error 38 us; skew 500 ppm
I20260923 17:39:32.101683 6912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:39:32.103879 6912 webserver.cc:460] Webserver started at http://127.20.240.66:45335/ using document root <none> and password file <none>
I20260923 17:39:32.104599 6912 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:32.104707 6912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:32.105012 6912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:32.108031 6912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "b523ac2e01c1459a8b7c52e2a8a25389"
format_stamp: "Formatted at 2026-09-23 17:39:32 on dist-test-slave-ppjl"
server_key: "74943f42cc482256d8c50b285b584e49"
server_key_iv: "7ec67b6e17d84b2b7ac39351761c1689"
server_key_version: "encryptionkey@0"
I20260923 17:39:32.108816 6912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "b523ac2e01c1459a8b7c52e2a8a25389"
format_stamp: "Formatted at 2026-09-23 17:39:32 on dist-test-slave-ppjl"
server_key: "74943f42cc482256d8c50b285b584e49"
server_key_iv: "7ec67b6e17d84b2b7ac39351761c1689"
server_key_version: "encryptionkey@0"
I20260923 17:39:32.113960 6912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:39:32.117513 6926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:32.118860 6912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:39:32.119009 6912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "b523ac2e01c1459a8b7c52e2a8a25389"
format_stamp: "Formatted at 2026-09-23 17:39:32 on dist-test-slave-ppjl"
server_key: "74943f42cc482256d8c50b285b584e49"
server_key_iv: "7ec67b6e17d84b2b7ac39351761c1689"
server_key_version: "encryptionkey@0"
I20260923 17:39:32.119246 6912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:32.137394 6912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:32.138485 6912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:32.138854 6912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:32.140231 6912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:32.142287 6912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:32.142380 6912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:32.142469 6912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:32.142524 6912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:32.172272 6912 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:41861
I20260923 17:39:32.172319 7038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:41861 every 8 connection(s)
I20260923 17:39:32.173601 6912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:39:32.178016 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 6912
I20260923 17:39:32.178205 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:39:32.178659 21441 external_mini_cluster.cc:1428] Setting key 5ebe1568e662087cf2ef210271726463
I20260923 17:39:32.181514 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41547
--builtin_ntp_servers=127.20.240.84:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:39:32.188776 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.176218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:32911 (local address 127.20.240.126:41547)
0923 17:39:32.176430 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:32.176436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:32.178065 (+ 1629us) server_negotiation.cc:407] Connection header received
0923 17:39:32.180617 (+ 2552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:32.180623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:32.180705 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:32.180809 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:32.182184 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.183284 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.185373 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.186010 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.186133 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.187875 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:32.187881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:32.187887 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:32.187910 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:32.187951 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:32.188036 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:32.188123 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:32.188356 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 17:39:32.188465 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:39:32.190264 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.176811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41547 (local address 127.20.240.66:32911)
0923 17:39:32.177875 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:32.177923 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:32.180273 (+ 2350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:32.181064 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:32.181076 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:32.181202 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:32.181989 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:32.182013 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.183463 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.183468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:32.185214 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:32.185229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.186245 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.186253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:32.186543 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.187506 (+ 963us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:39:32.187509 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:32.187560 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:39:32.187664 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:32.188134 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:32.188145 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:32.188152 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:39:32.188328 (+ 176us) client_negotiation.cc:240] Negotiation successful
0923 17:39:32.188726 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":275,"threads_started":1}
I20260923 17:39:32.191712 7039 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41547
I20260923 17:39:32.192025 7039 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:32.192780 7039 heartbeater.cc:511] Master 127.20.240.126:41547 requested a full tablet report, sending...
I20260923 17:39:32.194123 6722 ts_manager.cc:194] Registered new tserver with Master: b523ac2e01c1459a8b7c52e2a8a25389 (127.20.240.66:41861)
I20260923 17:39:32.194896 6722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.66:32911
W20260923 17:39:32.513094 7042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:32.513381 7042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:32.513489 7042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:32.520895 7042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:32.521001 7042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:32.521142 7042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:39:32.530586 7042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:32.532235 7042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:32.534062 7042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:32.546969 7047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:32.547030 7048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:32.547232 7050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:32.547703 7042 server_base.cc:1034] running on GCE node
I20260923 17:39:32.548306 7042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:32.549292 7042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:32.550499 7042 hybrid_clock.cc:648] HybridClock initialized: now 1790185172550445 us; error 46 us; skew 500 ppm
I20260923 17:39:32.550814 7042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:39:32.553241 7042 webserver.cc:460] Webserver started at http://127.20.240.67:33539/ using document root <none> and password file <none>
I20260923 17:39:32.553952 7042 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:32.554047 7042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:32.554356 7042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:32.557265 7042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "4cc7f224abc94b4c8cc59791242c7620"
format_stamp: "Formatted at 2026-09-23 17:39:32 on dist-test-slave-ppjl"
server_key: "01eef771942aa7f5f2f0a09b55d2afeb"
server_key_iv: "f309d93450d22b197d1c962277d9ae73"
server_key_version: "encryptionkey@0"
I20260923 17:39:32.558085 7042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "4cc7f224abc94b4c8cc59791242c7620"
format_stamp: "Formatted at 2026-09-23 17:39:32 on dist-test-slave-ppjl"
server_key: "01eef771942aa7f5f2f0a09b55d2afeb"
server_key_iv: "f309d93450d22b197d1c962277d9ae73"
server_key_version: "encryptionkey@0"
I20260923 17:39:32.563665 7042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:39:32.567481 7056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:32.568989 7042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:39:32.569154 7042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "4cc7f224abc94b4c8cc59791242c7620"
format_stamp: "Formatted at 2026-09-23 17:39:32 on dist-test-slave-ppjl"
server_key: "01eef771942aa7f5f2f0a09b55d2afeb"
server_key_iv: "f309d93450d22b197d1c962277d9ae73"
server_key_version: "encryptionkey@0"
I20260923 17:39:32.569384 7042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:32.598706 7042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:32.599682 7042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:32.599977 7042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:32.601032 7042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:32.602604 7042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:32.602699 7042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:32.602813 7042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:32.602880 7042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:32.633483 7042 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:39767
I20260923 17:39:32.633538 7168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:39767 every 8 connection(s)
I20260923 17:39:32.634734 7042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:39:32.642729 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 7042
I20260923 17:39:32.642937 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:39:32.643535 21441 external_mini_cluster.cc:1428] Setting key 2bc4dd5bbe008ddfd8da8ab17ff885c1
I20260923 17:39:32.648295 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.637420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:33461 (local address 127.20.240.126:41547)
0923 17:39:32.637627 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:32.637631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:32.639114 (+ 1483us) server_negotiation.cc:407] Connection header received
0923 17:39:32.641024 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:32.641029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:32.641141 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:32.641246 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:32.642542 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.643699 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.645475 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.645918 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.646029 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.647504 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:32.647508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:32.647512 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:32.647530 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:32.647563 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:32.647575 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:32.647654 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:32.647868 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 17:39:32.647968 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 17:39:32.649318 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.637948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41547 (local address 127.20.240.67:33461)
0923 17:39:32.638874 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:32.638989 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:32.640675 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:32.641502 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:32.641511 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:32.641614 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:32.642323 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:32.642347 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.643979 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.643986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:32.645320 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:32.645334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.646142 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.646155 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:32.646384 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.647137 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:39:32.647140 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:32.647225 (+ 85us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:39:32.647308 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:32.647660 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:32.647671 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:32.647680 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:39:32.647845 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 17:39:32.648282 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":276,"threads_started":1}
I20260923 17:39:32.650652 7169 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41547
I20260923 17:39:32.650978 7169 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:32.651731 7169 heartbeater.cc:511] Master 127.20.240.126:41547 requested a full tablet report, sending...
I20260923 17:39:32.653095 6722 ts_manager.cc:194] Registered new tserver with Master: 4cc7f224abc94b4c8cc59791242c7620 (127.20.240.67:39767)
I20260923 17:39:32.653867 6722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.67:33461
I20260923 17:39:32.658355 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:39:32.668859 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.662189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.20.240.126:41547)
0923 17:39:32.662420 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:32.662427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:32.662863 (+ 436us) server_negotiation.cc:407] Connection header received
0923 17:39:32.662967 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:32.662970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:32.663055 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:32.663199 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:32.663768 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.664480 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.666139 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.666580 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.666680 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.668097 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:32.668108 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:32.668113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:32.668140 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:32.668178 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:32.668195 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:32.668283 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:32.668455 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 17:39:32.668550 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:39:32.673422 6722 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:39:32.680823 6722 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:39:32.717456 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.694158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.20.240.65:40181)
0923 17:39:32.707838 (+ 13680us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:32.707844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:32.707872 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:39:32.707975 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:32.707982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:32.708272 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:32.709386 (+ 1114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:32.709648 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.711356 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.714702 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.716214 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.716344 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.716630 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:32.716895 (+ 265us) server_negotiation.cc:299] Negotiation successful
0923 17:39:32.717091 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13527,"thread_start_us":13391,"threads_started":1}
I20260923 17:39:32.717501 7182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.693862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:40181 (local address 127.0.0.1:34452)
0923 17:39:32.706053 (+ 12191us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:32.706126 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:32.706402 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:32.708597 (+ 2195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:32.708607 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:32.708650 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:32.709134 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:32.709149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.711550 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.711558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:32.714473 (+ 2915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:32.714495 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.716399 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.716408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:32.716614 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.716685 (+ 71us) client_negotiation.cc:769] Sending connection context
0923 17:39:32.716844 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 17:39:32.717061 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11771,"mutex_wait_us":100,"thread_start_us":13705,"threads_started":1}
I20260923 17:39:32.719693 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.694152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:39767 (local address 127.0.0.1:58902)
0923 17:39:32.707849 (+ 13697us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:32.707867 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:32.707991 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:32.709064 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:32.709070 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:32.709108 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:32.709504 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:32.709517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.711646 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.711652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:32.716343 (+ 4691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:32.716354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.718772 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.718779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:32.719032 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.719070 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 17:39:32.719151 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:39:32.719311 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13570,"mutex_wait_us":128,"thread_start_us":12495,"threads_started":1}
I20260923 17:39:32.720388 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.694339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58902 (local address 127.20.240.67:39767)
0923 17:39:32.707180 (+ 12841us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:32.707192 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:32.708135 (+ 943us) server_negotiation.cc:407] Connection header received
0923 17:39:32.708251 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:32.708261 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:32.708533 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:32.708734 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:32.709737 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.711449 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.716573 (+ 5124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.718218 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.718331 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.718630 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:32.719855 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0923 17:39:32.720058 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12593,"thread_start_us":12849,"threads_started":1}
I20260923 17:39:32.722244 6843 tablet_service.cc:1467] Processing CreateTablet for tablet e320cf8775664549b306576d3ef1cf81 (DEFAULT_TABLE table=test-table [id=66f7972bcead47aeb6787c994b67d005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:32.723018 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.708025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:41861 (local address 127.0.0.1:59628)
0923 17:39:32.714628 (+ 6603us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:32.714647 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:32.714784 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:32.715790 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:32.715795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:32.715823 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:32.716219 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:32.716230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.718099 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.718105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:32.720235 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:32.720253 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.722028 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.722036 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:32.722273 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.722309 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:39:32.722416 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 17:39:32.722559 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6455,"mutex_wait_us":1,"thread_start_us":6786,"threads_started":1}
I20260923 17:39:32.723068 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.708315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59628 (local address 127.20.240.66:41861)
0923 17:39:32.712822 (+ 4507us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:32.712829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:32.714769 (+ 1940us) server_negotiation.cc:407] Connection header received
0923 17:39:32.715009 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:32.715019 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:32.715330 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:32.715550 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:32.716431 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.717948 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.720652 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.721816 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.721967 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.722226 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:32.722481 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 17:39:32.722656 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4361,"thread_start_us":4237,"threads_started":1}
I20260923 17:39:32.724119 6843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e320cf8775664549b306576d3ef1cf81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:32.724890 7104 tablet_service.cc:1467] Processing CreateTablet for tablet e320cf8775664549b306576d3ef1cf81 (DEFAULT_TABLE table=test-table [id=66f7972bcead47aeb6787c994b67d005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:32.726487 7104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e320cf8775664549b306576d3ef1cf81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:32.727512 6974 tablet_service.cc:1467] Processing CreateTablet for tablet e320cf8775664549b306576d3ef1cf81 (DEFAULT_TABLE table=test-table [id=66f7972bcead47aeb6787c994b67d005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:32.728893 6974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e320cf8775664549b306576d3ef1cf81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:32.741271 7187 tablet_bootstrap.cc:492] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d: Bootstrap starting.
I20260923 17:39:32.743105 7188 tablet_bootstrap.cc:492] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620: Bootstrap starting.
I20260923 17:39:32.744581 7189 tablet_bootstrap.cc:492] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389: Bootstrap starting.
I20260923 17:39:32.747236 7188 tablet_bootstrap.cc:654] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:32.747232 7187 tablet_bootstrap.cc:654] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:32.747421 7189 tablet_bootstrap.cc:654] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:32.748373 7189 log.cc:824] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:32.748492 7188 log.cc:824] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:32.748492 7187 log.cc:824] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:32.751389 6908 heartbeater.cc:503] Master 127.20.240.126:41547 was elected leader, sending a full tablet report...
I20260923 17:39:32.751506 7189 tablet_bootstrap.cc:492] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389: No bootstrap required, opened a new log
I20260923 17:39:32.751783 7189 ts_tablet_manager.cc:1397] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260923 17:39:32.752140 7187 tablet_bootstrap.cc:492] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d: No bootstrap required, opened a new log
I20260923 17:39:32.752413 7187 ts_tablet_manager.cc:1397] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260923 17:39:32.753647 7188 tablet_bootstrap.cc:492] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620: No bootstrap required, opened a new log
I20260923 17:39:32.753950 7188 ts_tablet_manager.cc:1397] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260923 17:39:32.760550 7189 raft_consensus.cc:348] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc978b8f03bd43c7a5a16845044b104d" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40181 } } peers { permanent_uuid: "4cc7f224abc94b4c8cc59791242c7620" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39767 } } peers { permanent_uuid: "b523ac2e01c1459a8b7c52e2a8a25389" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 41861 } }
I20260923 17:39:32.761004 7189 raft_consensus.cc:374] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:32.761111 7189 raft_consensus.cc:729] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b523ac2e01c1459a8b7c52e2a8a25389, State: Initialized, Role: FOLLOWER
I20260923 17:39:32.761710 7189 consensus_queue.cc:260] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc978b8f03bd43c7a5a16845044b104d" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40181 } } peers { permanent_uuid: "4cc7f224abc94b4c8cc59791242c7620" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39767 } } peers { permanent_uuid: "b523ac2e01c1459a8b7c52e2a8a25389" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 41861 } }
I20260923 17:39:32.762836 7039 heartbeater.cc:503] Master 127.20.240.126:41547 was elected leader, sending a full tablet report...
I20260923 17:39:32.763550 7189 ts_tablet_manager.cc:1428] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260923 17:39:32.763593 7187 raft_consensus.cc:348] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc978b8f03bd43c7a5a16845044b104d" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40181 } } peers { permanent_uuid: "4cc7f224abc94b4c8cc59791242c7620" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39767 } } peers { permanent_uuid: "b523ac2e01c1459a8b7c52e2a8a25389" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 41861 } }
I20260923 17:39:32.764076 7187 raft_consensus.cc:374] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:32.764173 7187 raft_consensus.cc:729] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc978b8f03bd43c7a5a16845044b104d, State: Initialized, Role: FOLLOWER
I20260923 17:39:32.764303 7188 raft_consensus.cc:348] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc978b8f03bd43c7a5a16845044b104d" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40181 } } peers { permanent_uuid: "4cc7f224abc94b4c8cc59791242c7620" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39767 } } peers { permanent_uuid: "b523ac2e01c1459a8b7c52e2a8a25389" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 41861 } }
I20260923 17:39:32.764752 7188 raft_consensus.cc:374] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:32.764868 7188 raft_consensus.cc:729] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc7f224abc94b4c8cc59791242c7620, State: Initialized, Role: FOLLOWER
I20260923 17:39:32.764758 7187 consensus_queue.cc:260] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc978b8f03bd43c7a5a16845044b104d" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40181 } } peers { permanent_uuid: "4cc7f224abc94b4c8cc59791242c7620" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39767 } } peers { permanent_uuid: "b523ac2e01c1459a8b7c52e2a8a25389" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 41861 } }
I20260923 17:39:32.765475 7188 consensus_queue.cc:260] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc978b8f03bd43c7a5a16845044b104d" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40181 } } peers { permanent_uuid: "4cc7f224abc94b4c8cc59791242c7620" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39767 } } peers { permanent_uuid: "b523ac2e01c1459a8b7c52e2a8a25389" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 41861 } }
I20260923 17:39:32.767409 7188 ts_tablet_manager.cc:1428] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260923 17:39:32.768098 7187 ts_tablet_manager.cc:1428] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260923 17:39:32.768261 7169 heartbeater.cc:503] Master 127.20.240.126:41547 was elected leader, sending a full tablet report...
I20260923 17:39:32.814049 7195 raft_consensus.cc:482] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:39:32.814317 7195 raft_consensus.cc:504] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc978b8f03bd43c7a5a16845044b104d" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40181 } } peers { permanent_uuid: "4cc7f224abc94b4c8cc59791242c7620" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39767 } } peers { permanent_uuid: "b523ac2e01c1459a8b7c52e2a8a25389" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 41861 } }
I20260923 17:39:32.816133 7195 leader_election.cc:288] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc978b8f03bd43c7a5a16845044b104d (127.20.240.65:40181), b523ac2e01c1459a8b7c52e2a8a25389 (127.20.240.66:41861)
I20260923 17:39:32.822898 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.816594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:40599 (local address 127.20.240.65:40181)
0923 17:39:32.816828 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:32.816835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:32.816856 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:39:32.816937 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:32.816942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:32.817127 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:32.817231 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:32.817757 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.818723 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.820872 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.821889 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.822057 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.822123 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:32.822495 (+ 372us) server_negotiation.cc:299] Negotiation successful
0923 17:39:32.822613 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:39:32.822999 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.816394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:40181 (local address 127.20.240.67:40599)
0923 17:39:32.816611 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:32.816631 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:32.816756 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:32.817430 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:32.817434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:32.817459 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:32.817615 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:32.817622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.818868 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.818874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:32.820659 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:32.820688 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.822112 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.822119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:32.822336 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.822373 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 17:39:32.822475 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 17:39:32.822652 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":33}
I20260923 17:39:32.823536 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.817166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:53377 (local address 127.20.240.66:41861)
0923 17:39:32.817361 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:32.817366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:32.817614 (+ 248us) server_negotiation.cc:407] Connection header received
0923 17:39:32.817823 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:32.817827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:32.817897 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:32.817966 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:32.818565 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.819579 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.821503 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.822533 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.822679 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.822745 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:32.823116 (+ 371us) server_negotiation.cc:299] Negotiation successful
0923 17:39:32.823276 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:39:32.824151 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.816877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:41861 (local address 127.20.240.67:53377)
0923 17:39:32.817512 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:32.817526 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:32.817661 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:32.818004 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:32.818009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:32.818035 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:32.818381 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:32.818393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.819723 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.819727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:32.821284 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:32.821301 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.822731 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.822738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:32.822912 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.822948 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:39:32.823767 (+ 819us) client_negotiation.cc:240] Negotiation successful
0923 17:39:32.823901 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":20,"thread_start_us":309,"threads_started":1}
I20260923 17:39:32.824306 6863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e320cf8775664549b306576d3ef1cf81" candidate_uuid: "4cc7f224abc94b4c8cc59791242c7620" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc978b8f03bd43c7a5a16845044b104d" is_pre_election: true
I20260923 17:39:32.824944 6863 raft_consensus.cc:2454] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc7f224abc94b4c8cc59791242c7620 in term 0.
I20260923 17:39:32.825043 6994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e320cf8775664549b306576d3ef1cf81" candidate_uuid: "4cc7f224abc94b4c8cc59791242c7620" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523ac2e01c1459a8b7c52e2a8a25389" is_pre_election: true
I20260923 17:39:32.825548 6994 raft_consensus.cc:2454] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cc7f224abc94b4c8cc59791242c7620 in term 0.
I20260923 17:39:32.825598 7058 leader_election.cc:302] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc7f224abc94b4c8cc59791242c7620, fc978b8f03bd43c7a5a16845044b104d; no voters:
I20260923 17:39:32.826000 7195 raft_consensus.cc:2781] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:39:32.826162 7195 raft_consensus.cc:482] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:39:32.826272 7195 raft_consensus.cc:3037] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:32.828800 7195 raft_consensus.cc:504] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc978b8f03bd43c7a5a16845044b104d" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40181 } } peers { permanent_uuid: "4cc7f224abc94b4c8cc59791242c7620" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39767 } } peers { permanent_uuid: "b523ac2e01c1459a8b7c52e2a8a25389" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 41861 } }
I20260923 17:39:32.829419 7195 leader_election.cc:288] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [CANDIDATE]: Term 1 election: Requested vote from peers fc978b8f03bd43c7a5a16845044b104d (127.20.240.65:40181), b523ac2e01c1459a8b7c52e2a8a25389 (127.20.240.66:41861)
I20260923 17:39:32.829993 6863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e320cf8775664549b306576d3ef1cf81" candidate_uuid: "4cc7f224abc94b4c8cc59791242c7620" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc978b8f03bd43c7a5a16845044b104d"
I20260923 17:39:32.830113 6994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e320cf8775664549b306576d3ef1cf81" candidate_uuid: "4cc7f224abc94b4c8cc59791242c7620" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b523ac2e01c1459a8b7c52e2a8a25389"
I20260923 17:39:32.830229 6863 raft_consensus.cc:3037] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:32.830329 6994 raft_consensus.cc:3037] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:32.832753 6994 raft_consensus.cc:2454] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc7f224abc94b4c8cc59791242c7620 in term 1.
I20260923 17:39:32.833040 6863 raft_consensus.cc:2454] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cc7f224abc94b4c8cc59791242c7620 in term 1.
I20260923 17:39:32.833448 7058 leader_election.cc:302] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cc7f224abc94b4c8cc59791242c7620, b523ac2e01c1459a8b7c52e2a8a25389; no voters:
I20260923 17:39:32.833876 7195 raft_consensus.cc:2781] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:32.834420 7195 raft_consensus.cc:686] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [term 1 LEADER]: Becoming Leader. State: Replica: 4cc7f224abc94b4c8cc59791242c7620, State: Running, Role: LEADER
I20260923 17:39:32.834971 7195 consensus_queue.cc:237] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc978b8f03bd43c7a5a16845044b104d" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40181 } } peers { permanent_uuid: "4cc7f224abc94b4c8cc59791242c7620" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39767 } } peers { permanent_uuid: "b523ac2e01c1459a8b7c52e2a8a25389" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 41861 } }
I20260923 17:39:32.840215 6720 catalog_manager.cc:5495] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cc7f224abc94b4c8cc59791242c7620 (127.20.240.67). New cstate: current_term: 1 leader_uuid: "4cc7f224abc94b4c8cc59791242c7620" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc978b8f03bd43c7a5a16845044b104d" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cc7f224abc94b4c8cc59791242c7620" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 39767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b523ac2e01c1459a8b7c52e2a8a25389" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 41861 } health_report { overall_health: UNKNOWN } } }
I20260923 17:39:32.865190 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.859699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.20.240.67:39767)
0923 17:39:32.859931 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:32.859941 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:32.859961 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:39:32.860323 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:32.860330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:32.860416 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:32.860545 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:39:32.860926 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.861703 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.863141 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.863576 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.863671 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.864183 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:39:32.864694 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:39:32.864800 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:32.864899 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 17:39:32.864973 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 17:39:32.873791 6863 raft_consensus.cc:1261] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d [term 1 FOLLOWER]: Refusing update from remote peer 4cc7f224abc94b4c8cc59791242c7620: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:39:32.873790 6994 raft_consensus.cc:1261] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389 [term 1 FOLLOWER]: Refusing update from remote peer 4cc7f224abc94b4c8cc59791242c7620: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:39:32.874701 7197 consensus_queue.cc:1035] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b523ac2e01c1459a8b7c52e2a8a25389" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 41861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:39:32.875057 7195 consensus_queue.cc:1035] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc978b8f03bd43c7a5a16845044b104d" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 40181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:39:32.886247 7200 mvcc.cc:205] Tried to move back new op lower bound from 7332598468079796224 to 7332598467936817152. Current Snapshot: MvccSnapshot[applied={T|T < 7332598468079796224}]
I20260923 17:39:32.890542 7202 mvcc.cc:205] Tried to move back new op lower bound from 7332598468079796224 to 7332598467936817152. Current Snapshot: MvccSnapshot[applied={T|T < 7332598468079796224}]
I20260923 17:39:32.903962 6722 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:39:32.904251 6722 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:39:32.908113 6722 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b828f366d4e414da2b4ea006f6bd00b: Sending DeleteTablet for 3 replicas of tablet e320cf8775664549b306576d3ef1cf81
I20260923 17:39:32.909238 7104 tablet_service.cc:1514] Processing DeleteTablet for tablet e320cf8775664549b306576d3ef1cf81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:32 UTC) from {username='slave'} at 127.0.0.1:58902
I20260923 17:39:32.909366 6974 tablet_service.cc:1514] Processing DeleteTablet for tablet e320cf8775664549b306576d3ef1cf81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:32 UTC) from {username='slave'} at 127.0.0.1:59628
I20260923 17:39:32.909528 6843 tablet_service.cc:1514] Processing DeleteTablet for tablet e320cf8775664549b306576d3ef1cf81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:32 UTC) from {username='slave'} at 127.0.0.1:34452
I20260923 17:39:32.910249 7210 tablet_replica.cc:330] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620: stopping tablet replica
I20260923 17:39:32.910795 7210 raft_consensus.cc:2229] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:39:32.910902 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 6781
I20260923 17:39:32.911701 7210 raft_consensus.cc:2258] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:32.913928 7210 ts_tablet_manager.cc:1905] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:32.921000 7212 tablet_replica.cc:330] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d: stopping tablet replica
I20260923 17:39:32.921561 7212 raft_consensus.cc:2229] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:39:32.922137 7212 raft_consensus.cc:2258] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:32.922468 7210 ts_tablet_manager.cc:1918] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:32.922724 7210 log.cc:1196] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/e320cf8775664549b306576d3ef1cf81
I20260923 17:39:32.923434 7210 ts_tablet_manager.cc:1939] T e320cf8775664549b306576d3ef1cf81 P 4cc7f224abc94b4c8cc59791242c7620: Deleting consensus metadata
I20260923 17:39:32.925537 6709 catalog_manager.cc:4841] TS 4cc7f224abc94b4c8cc59791242c7620 (127.20.240.67:39767): tablet e320cf8775664549b306576d3ef1cf81 (table test-table [id=66f7972bcead47aeb6787c994b67d005]) successfully deleted
I20260923 17:39:32.927824 7211 tablet_replica.cc:330] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389: stopping tablet replica
I20260923 17:39:32.928129 7212 ts_tablet_manager.cc:1905] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:32.928268 7211 raft_consensus.cc:2229] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:39:32.928758 7211 raft_consensus.cc:2258] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:32.931879 7211 ts_tablet_manager.cc:1905] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:32.932498 7183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:32.912220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.20.240.65:40181)
0923 17:39:32.912416 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:32.912422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:32.922966 (+ 10544us) server_negotiation.cc:407] Connection header received
0923 17:39:32.923294 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:32.923300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:32.923413 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:32.923498 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:32.924197 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.925320 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.927415 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:32.928189 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:32.928321 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:32.930624 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:32.930637 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:32.930655 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:32.931498 (+ 843us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:32.931575 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:32.931603 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:32.931719 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:32.931942 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 17:39:32.932069 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:39:32.934005 6903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:32.941080 7211 ts_tablet_manager.cc:1918] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:32.941263 7211 log.cc:1196] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/e320cf8775664549b306576d3ef1cf81
I20260923 17:39:32.941876 7211 ts_tablet_manager.cc:1939] T e320cf8775664549b306576d3ef1cf81 P b523ac2e01c1459a8b7c52e2a8a25389: Deleting consensus metadata
I20260923 17:39:32.943666 6707 catalog_manager.cc:4841] TS b523ac2e01c1459a8b7c52e2a8a25389 (127.20.240.66:41861): tablet e320cf8775664549b306576d3ef1cf81 (table test-table [id=66f7972bcead47aeb6787c994b67d005]) successfully deleted
I20260923 17:39:33.045992 7212 ts_tablet_manager.cc:1918] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:33.046319 7212 log.cc:1196] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/e320cf8775664549b306576d3ef1cf81
I20260923 17:39:33.046803 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 6781
I20260923 17:39:33.046995 7212 ts_tablet_manager.cc:1939] T e320cf8775664549b306576d3ef1cf81 P fc978b8f03bd43c7a5a16845044b104d: Deleting consensus metadata
W20260923 17:39:33.068099 6707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.240.65:40181 (error 108)
I20260923 17:39:33.068789 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 6912
I20260923 17:39:33.070298 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:33.069600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:40181 (local address 127.0.0.1:34478)
0923 17:39:33.070000 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:33.070139 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
W20260923 17:39:33.070570 6707 catalog_manager.cc:4584] TS fc978b8f03bd43c7a5a16845044b104d (127.20.240.65:40181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e320cf8775664549b306576d3ef1cf81: Network error: Client connection negotiation failed: client connection to 127.20.240.65:40181: connect: Connection refused (error 111)
I20260923 17:39:33.076934 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:33.069672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.20.240.66:41861)
0923 17:39:33.069850 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:33.069856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:33.070002 (+ 146us) server_negotiation.cc:407] Connection header received
0923 17:39:33.070184 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:33.070190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:33.070322 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:33.070426 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:33.071018 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:33.071828 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:33.073785 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:33.074234 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:33.074333 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:33.075591 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:33.075601 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:33.075615 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:33.076191 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:33.076240 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:33.076266 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:33.076352 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:33.076497 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 17:39:33.076605 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:39:33.077600 7034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:33.124238 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:33.123676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:40181 (local address 127.0.0.1:34484)
0923 17:39:33.123914 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:33.124037 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 17:39:33.125425 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:33.125023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:40181 (local address 127.0.0.1:34500)
0923 17:39:33.125179 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:33.125265 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 17:39:33.167189 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 6912
I20260923 17:39:33.182088 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 7042
I20260923 17:39:33.185809 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:33.185219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:40181 (local address 127.0.0.1:34512)
0923 17:39:33.185509 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:33.185612 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260923 17:39:33.186914 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:33.186491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:40181 (local address 127.0.0.1:34526)
0923 17:39:33.186697 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:33.186779 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260923 17:39:33.190137 7184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:33.182909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.20.240.67:39767)
0923 17:39:33.183070 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:33.183075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:33.183093 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:39:33.183271 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:33.183275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:33.183359 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:33.183449 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:33.183894 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:33.184756 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:33.186739 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:33.187409 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:33.187538 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:33.188585 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:33.188595 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:33.188609 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:33.189251 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:33.189299 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:33.189320 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:33.189418 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:33.189563 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 17:39:33.189678 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:39:33.190806 7164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:33.267748 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:33.267242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:40181 (local address 127.0.0.1:34536)
0923 17:39:33.267478 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:33.267580 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 17:39:33.268821 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:33.268430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:40181 (local address 127.0.0.1:34540)
0923 17:39:33.268635 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:33.268706 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:40181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260923 17:39:33.312129 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 7042
I20260923 17:39:33.328116 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 6691
I20260923 17:39:33.329264 6752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:33.438868 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 6691
2026-09-23T17:39:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2770 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:39:33 dist-test-slave-ppjl krb5kdc[7221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:39:33 dist-test-slave-ppjl krb5kdc[7221](info): set up 2 sockets
Sep 23 17:39:33 dist-test-slave-ppjl krb5kdc[7221](info): commencing operation
krb5kdc: starting...
W20260923 17:39:34.464023 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:39:34 dist-test-slave-ppjl krb5kdc[7221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:39:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:39:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:39:34.711750 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:36485
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:43055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:36485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:35.050820 7234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:35.051128 7234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:35.051268 7234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:35.058887 7234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:39:35.059005 7234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:35.059054 7234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:35.059134 7234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:39:35.059233 7234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:39:35.069710 7234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:43055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:36485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:36485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:35.071110 7234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:35.072479 7234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:35.085258 7239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:35.085260 7242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:35.085247 7240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:35.086073 7234 server_base.cc:1034] running on GCE node
I20260923 17:39:35.086921 7234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:35.088553 7234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:35.089805 7234 hybrid_clock.cc:648] HybridClock initialized: now 1790185175089755 us; error 38 us; skew 500 ppm
I20260923 17:39:35.090265 7234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:35 dist-test-slave-ppjl krb5kdc[7221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185175, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:35.093838 7234 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:39:35.095585 7234 webserver.cc:460] Webserver started at http://127.20.240.126:37029/ using document root <none> and password file <none>
I20260923 17:39:35.096339 7234 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:35.096479 7234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:35.096791 7234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:35.100122 7234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "e629591aef044ceeb04c7e8feb4af58b"
format_stamp: "Formatted at 2026-09-23 17:39:35 on dist-test-slave-ppjl"
server_key: "73e44daea38fc6ba4c9597cd60b10f59"
server_key_iv: "72d365442fcc2b814c56f8d6aa6f5648"
server_key_version: "encryptionkey@0"
I20260923 17:39:35.100956 7234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "e629591aef044ceeb04c7e8feb4af58b"
format_stamp: "Formatted at 2026-09-23 17:39:35 on dist-test-slave-ppjl"
server_key: "73e44daea38fc6ba4c9597cd60b10f59"
server_key_iv: "72d365442fcc2b814c56f8d6aa6f5648"
server_key_version: "encryptionkey@0"
I20260923 17:39:35.106673 7234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:39:35.110625 7249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:35.112262 7234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:39:35.112459 7234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "e629591aef044ceeb04c7e8feb4af58b"
format_stamp: "Formatted at 2026-09-23 17:39:35 on dist-test-slave-ppjl"
server_key: "73e44daea38fc6ba4c9597cd60b10f59"
server_key_iv: "72d365442fcc2b814c56f8d6aa6f5648"
server_key_version: "encryptionkey@0"
I20260923 17:39:35.112661 7234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:35.133419 7234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:35.138130 7234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:35.138417 7234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:35.156538 7234 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:36485
I20260923 17:39:35.156556 7300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:36485 every 8 connection(s)
I20260923 17:39:35.157924 7234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:39:35.161867 7301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:35.165848 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 7234
I20260923 17:39:35.166024 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:39:35.166488 21441 external_mini_cluster.cc:1428] Setting key 59ce678489a5ec9066bfbde74a9b2573
I20260923 17:39:35.175696 7301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b: Bootstrap starting.
I20260923 17:39:35.179937 7301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:35.181334 7301 log.cc:824] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:39:35 dist-test-slave-ppjl krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:39:35.184749 7301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b: No bootstrap required, opened a new log
I20260923 17:39:35.191020 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:35.168427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36000 (local address 127.20.240.126:36485)
0923 17:39:35.169267 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:35.169279 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:35.169320 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:39:35.170994 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:35.171019 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:35.171508 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:35.171864 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:35.174090 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:35.175522 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:35.177442 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:35.178312 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:35.178490 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:35.183493 (+ 5003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:35.183502 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:35.183518 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:35.183562 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:35.186732 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:35.187603 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:35.187611 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:35.187618 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:35.187800 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:35.188184 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:35.188192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:35.188194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:35.188733 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:35.189071 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:35.189477 (+ 406us) server_negotiation.cc:299] Negotiation successful
0923 17:39:35.189946 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":370,"threads_started":1}
I20260923 17:39:35.193240 7301 raft_consensus.cc:348] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e629591aef044ceeb04c7e8feb4af58b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36485 } }
I20260923 17:39:35.193625 7301 raft_consensus.cc:374] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:35.193732 7301 raft_consensus.cc:729] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e629591aef044ceeb04c7e8feb4af58b, State: Initialized, Role: FOLLOWER
I20260923 17:39:35.194365 7301 consensus_queue.cc:260] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e629591aef044ceeb04c7e8feb4af58b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36485 } }
I20260923 17:39:35.194599 7301 raft_consensus.cc:388] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:39:35.194720 7301 raft_consensus.cc:482] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:39:35.194877 7301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:35.197567 7301 raft_consensus.cc:504] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e629591aef044ceeb04c7e8feb4af58b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36485 } }
I20260923 17:39:35.198104 7301 leader_election.cc:302] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e629591aef044ceeb04c7e8feb4af58b; no voters:
I20260923 17:39:35.198693 7301 leader_election.cc:288] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:39:35.198815 7306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:35.199517 7306 raft_consensus.cc:686] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [term 1 LEADER]: Becoming Leader. State: Replica: e629591aef044ceeb04c7e8feb4af58b, State: Running, Role: LEADER
I20260923 17:39:35.200055 7306 consensus_queue.cc:237] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e629591aef044ceeb04c7e8feb4af58b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36485 } }
I20260923 17:39:35.200778 7301 sys_catalog.cc:564] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:39:35.202895 7307 sys_catalog.cc:455] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e629591aef044ceeb04c7e8feb4af58b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e629591aef044ceeb04c7e8feb4af58b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36485 } } }
I20260923 17:39:35.202888 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e629591aef044ceeb04c7e8feb4af58b. Latest consensus state: current_term: 1 leader_uuid: "e629591aef044ceeb04c7e8feb4af58b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e629591aef044ceeb04c7e8feb4af58b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 36485 } } }
I20260923 17:39:35.203373 7307 sys_catalog.cc:458] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:35.203373 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:35.208002 7311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:39:35.212355 7311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:39:35.221356 7311 catalog_manager.cc:1347] Generated new cluster ID: 80a48fcc6b014b79bab06b4912c6ab08
I20260923 17:39:35.221485 7311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:39:35.232313 7311 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:39:35.234220 7311 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:39:35.244620 7311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b: Generated new TSK 0
I20260923 17:39:35.245704 7311 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:39:35.311265 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:36485
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:43055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:35.644698 7327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:35.645006 7327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:35.645144 7327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:35.652505 7327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:35.652642 7327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:35.652796 7327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:39:35.662659 7327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:43055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:36485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:35.664000 7327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:35.665328 7327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:35.678872 7332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:35.678799 7335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:35.678916 7333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:35.680459 7327 server_base.cc:1034] running on GCE node
I20260923 17:39:35.681155 7327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:35.682183 7327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:35.683416 7327 hybrid_clock.cc:648] HybridClock initialized: now 1790185175683360 us; error 42 us; skew 500 ppm
I20260923 17:39:35.683764 7327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:35 dist-test-slave-ppjl krb5kdc[7221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185175, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:35.686986 7327 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:39:35.688627 7327 webserver.cc:460] Webserver started at http://127.20.240.65:39909/ using document root <none> and password file <none>
I20260923 17:39:35.689319 7327 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:35.689440 7327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:35.689751 7327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:35.692950 7327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "05a4acdb0e614082a963ab4160f04fa3"
format_stamp: "Formatted at 2026-09-23 17:39:35 on dist-test-slave-ppjl"
server_key: "02fb5fcfb23f6669712411ff507a2946"
server_key_iv: "ecf8786f8fd78ec2221814e4a0a571bd"
server_key_version: "encryptionkey@0"
I20260923 17:39:35.693703 7327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "05a4acdb0e614082a963ab4160f04fa3"
format_stamp: "Formatted at 2026-09-23 17:39:35 on dist-test-slave-ppjl"
server_key: "02fb5fcfb23f6669712411ff507a2946"
server_key_iv: "ecf8786f8fd78ec2221814e4a0a571bd"
server_key_version: "encryptionkey@0"
I20260923 17:39:35.699076 7327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 17:39:35.702788 7342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:35.704308 7327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 17:39:35.704500 7327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "05a4acdb0e614082a963ab4160f04fa3"
format_stamp: "Formatted at 2026-09-23 17:39:35 on dist-test-slave-ppjl"
server_key: "02fb5fcfb23f6669712411ff507a2946"
server_key_iv: "ecf8786f8fd78ec2221814e4a0a571bd"
server_key_version: "encryptionkey@0"
I20260923 17:39:35.704761 7327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:35.724565 7327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:35.729030 7327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:35.729291 7327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:35.730228 7327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:35.731590 7327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:35.731662 7327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:35.731755 7327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:35.731796 7327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:35.762787 7327 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:37401
I20260923 17:39:35.762832 7454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:37401 every 8 connection(s)
I20260923 17:39:35.764228 7327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:39:35.768752 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 7327
I20260923 17:39:35.768942 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:39:35.769425 21441 external_mini_cluster.cc:1428] Setting key 28d175e598154c435b0e3bd57a50036c
Sep 23 17:39:35 dist-test-slave-ppjl krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185175, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:39:35.784880 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:35.766964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:36793 (local address 127.20.240.126:36485)
0923 17:39:35.767438 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:35.767443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:35.768445 (+ 1002us) server_negotiation.cc:407] Connection header received
0923 17:39:35.770375 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:35.770381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:35.770487 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:35.770583 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:35.772205 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:35.772930 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:35.774515 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:35.774913 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:35.775037 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:35.778422 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:35.778428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:35.778508 (+ 80us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:35.778546 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:35.781692 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:35.782519 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:35.782525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:35.782528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:35.782677 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:35.783081 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:35.783085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:35.783088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:35.783433 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:35.783568 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:35.784307 (+ 739us) server_negotiation.cc:299] Negotiation successful
0923 17:39:35.784537 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":252,"threads_started":1}
I20260923 17:39:35.785864 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:35.767594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36485 (local address 127.20.240.65:36793)
0923 17:39:35.768282 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:35.768326 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:35.770110 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:35.770843 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:35.770852 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:35.771433 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:35.772028 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:35.772046 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:35.773102 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:35.773106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:35.774362 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:35.774376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:35.775177 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:35.775186 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:35.775412 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:35.776046 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:39:35.776074 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:35.778196 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:35.781874 (+ 3678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:35.781883 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:35.781898 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:35.782387 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:35.782769 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:35.782772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:35.782775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:35.782974 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:35.783569 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:35.783576 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:35.783925 (+ 349us) client_negotiation.cc:769] Sending connection context
0923 17:39:35.784162 (+ 237us) client_negotiation.cc:240] Negotiation successful
0923 17:39:35.784628 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":259,"threads_started":1}
I20260923 17:39:35.787533 7455 heartbeater.cc:347] Connected to a master server at 127.20.240.126:36485
I20260923 17:39:35.787859 7455 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:35.788748 7455 heartbeater.cc:511] Master 127.20.240.126:36485 requested a full tablet report, sending...
I20260923 17:39:35.791219 7266 ts_manager.cc:194] Registered new tserver with Master: 05a4acdb0e614082a963ab4160f04fa3 (127.20.240.65:37401)
I20260923 17:39:35.792768 7266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:36793
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:39:35.826202 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:36485
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:43055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:36.156476 7461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:36.156745 7461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:36.156869 7461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:36.164144 7461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:36.164252 7461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:36.164402 7461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:39:36.174114 7461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:43055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:36485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:36.175441 7461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:36.176740 7461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:36.189769 7466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:36.189785 7467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:36.190466 7469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:36.190622 7461 server_base.cc:1034] running on GCE node
I20260923 17:39:36.191293 7461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:36.192216 7461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:36.193410 7461 hybrid_clock.cc:648] HybridClock initialized: now 1790185176193359 us; error 45 us; skew 500 ppm
I20260923 17:39:36.193714 7461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:36 dist-test-slave-ppjl krb5kdc[7221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185176, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:36.202152 7461 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:39:36.203685 7461 webserver.cc:460] Webserver started at http://127.20.240.66:45175/ using document root <none> and password file <none>
I20260923 17:39:36.204367 7461 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:36.204471 7461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:36.204751 7461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:36.207808 7461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c"
format_stamp: "Formatted at 2026-09-23 17:39:36 on dist-test-slave-ppjl"
server_key: "46b4e8dded11e661a29fb79298a2332e"
server_key_iv: "944dbc2c58925bbe80529544915e1064"
server_key_version: "encryptionkey@0"
I20260923 17:39:36.208702 7461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c"
format_stamp: "Formatted at 2026-09-23 17:39:36 on dist-test-slave-ppjl"
server_key: "46b4e8dded11e661a29fb79298a2332e"
server_key_iv: "944dbc2c58925bbe80529544915e1064"
server_key_version: "encryptionkey@0"
I20260923 17:39:36.214326 7461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:39:36.218019 7476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:36.219592 7461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:39:36.219743 7461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c"
format_stamp: "Formatted at 2026-09-23 17:39:36 on dist-test-slave-ppjl"
server_key: "46b4e8dded11e661a29fb79298a2332e"
server_key_iv: "944dbc2c58925bbe80529544915e1064"
server_key_version: "encryptionkey@0"
I20260923 17:39:36.219916 7461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:36.234798 7461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:36.238759 7461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:36.238996 7461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:36.239986 7461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:36.241276 7461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:36.241348 7461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:36.241438 7461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:36.241483 7461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:36.271272 7461 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:45643
I20260923 17:39:36.271318 7588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:45643 every 8 connection(s)
I20260923 17:39:36.272641 7461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:39:36.279781 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 7461
I20260923 17:39:36.280030 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:39:36.280527 21441 external_mini_cluster.cc:1428] Setting key 6c9ec2f7c73bcc4b88b59db8b2881904
Sep 23 17:39:36 dist-test-slave-ppjl krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185176, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:39:36.293591 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.275350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:49475 (local address 127.20.240.126:36485)
0923 17:39:36.275568 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:36.275575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:36.276904 (+ 1329us) server_negotiation.cc:407] Connection header received
0923 17:39:36.278963 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:36.278967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:36.279052 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:36.279145 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:36.281028 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.281798 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.283478 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.283891 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.284003 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.287546 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:36.287560 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:36.287565 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:36.287607 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:36.290430 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:36.291226 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:36.291230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:36.291233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:36.291366 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:36.291809 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:36.291813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:36.291815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:36.292113 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:36.292249 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:36.293013 (+ 764us) server_negotiation.cc:299] Negotiation successful
0923 17:39:36.293257 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:39:36.294605 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.275947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36485 (local address 127.20.240.66:49475)
0923 17:39:36.276729 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:36.276768 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:36.278652 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:36.279429 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:36.279439 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:36.280031 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:36.280785 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:36.280807 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.281982 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.281987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:36.283325 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:36.283341 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.284117 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.284125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:36.284351 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.285018 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:39:36.285043 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:36.287268 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:36.290597 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:36.290605 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:36.290620 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:36.291067 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:36.291481 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:36.291484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:36.291487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:36.291688 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:36.292243 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:36.292251 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:36.292624 (+ 373us) client_negotiation.cc:769] Sending connection context
0923 17:39:36.292865 (+ 241us) client_negotiation.cc:240] Negotiation successful
0923 17:39:36.293445 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":288,"threads_started":1}
I20260923 17:39:36.296180 7589 heartbeater.cc:347] Connected to a master server at 127.20.240.126:36485
I20260923 17:39:36.296489 7589 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:36.297212 7589 heartbeater.cc:511] Master 127.20.240.126:36485 requested a full tablet report, sending...
I20260923 17:39:36.298765 7266 ts_manager.cc:194] Registered new tserver with Master: 0c2130a9a8ed4b3a8d1524dab61bf10c (127.20.240.66:45643)
I20260923 17:39:36.299885 7266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:49475
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:39:36.332643 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:36485
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:43055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:36.656276 7594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:36.656574 7594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:36.656669 7594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:36.664011 7594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:36.664134 7594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:36.664247 7594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:39:36.674223 7594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:43055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:36485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:36.675704 7594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:36.677016 7594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:36.691184 7602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:36.691219 7600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:36.691249 7599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:36.691968 7594 server_base.cc:1034] running on GCE node
I20260923 17:39:36.692628 7594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:36.693647 7594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:36.694857 7594 hybrid_clock.cc:648] HybridClock initialized: now 1790185176694803 us; error 50 us; skew 500 ppm
I20260923 17:39:36.695216 7594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:36 dist-test-slave-ppjl krb5kdc[7221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185176, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:36.698518 7594 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:39:36.700255 7594 webserver.cc:460] Webserver started at http://127.20.240.67:46463/ using document root <none> and password file <none>
I20260923 17:39:36.701027 7594 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:36.701150 7594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:36.701460 7594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:36.704571 7594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "095b882d941b4cb0925e535a7e44c7d3"
format_stamp: "Formatted at 2026-09-23 17:39:36 on dist-test-slave-ppjl"
server_key: "f2b348720a17846a37049082801e5dda"
server_key_iv: "e218cbfd44551c7ffb41227968b6a242"
server_key_version: "encryptionkey@0"
I20260923 17:39:36.705390 7594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "095b882d941b4cb0925e535a7e44c7d3"
format_stamp: "Formatted at 2026-09-23 17:39:36 on dist-test-slave-ppjl"
server_key: "f2b348720a17846a37049082801e5dda"
server_key_iv: "e218cbfd44551c7ffb41227968b6a242"
server_key_version: "encryptionkey@0"
I20260923 17:39:36.711077 7594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:39:36.714780 7609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:36.716419 7594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:39:36.716591 7594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "095b882d941b4cb0925e535a7e44c7d3"
format_stamp: "Formatted at 2026-09-23 17:39:36 on dist-test-slave-ppjl"
server_key: "f2b348720a17846a37049082801e5dda"
server_key_iv: "e218cbfd44551c7ffb41227968b6a242"
server_key_version: "encryptionkey@0"
I20260923 17:39:36.716810 7594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:36.738296 7594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:36.742822 7594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:36.743214 7594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:36.744225 7594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:36.745608 7594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:36.745687 7594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:36.745786 7594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:36.745839 7594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:36.777675 7594 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:43943
I20260923 17:39:36.777750 7722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:43943 every 8 connection(s)
I20260923 17:39:36.779206 7594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:39:36.786466 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 7594
I20260923 17:39:36.786676 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:39:36.787230 21441 external_mini_cluster.cc:1428] Setting key d8996258203dae401d2ebaa8aa3477f0
Sep 23 17:39:36 dist-test-slave-ppjl krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185176, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:39:36.796176 7455 heartbeater.cc:503] Master 127.20.240.126:36485 was elected leader, sending a full tablet report...
I20260923 17:39:36.800623 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.781765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:56187 (local address 127.20.240.126:36485)
0923 17:39:36.781977 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:36.781984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:36.783109 (+ 1125us) server_negotiation.cc:407] Connection header received
0923 17:39:36.784929 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:36.784935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:36.785056 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:36.785157 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:36.786952 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.787900 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.789985 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.790501 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.790603 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.794342 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:36.794349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:36.794353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:36.794402 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:36.797366 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:36.798263 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:36.798267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:36.798270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:36.798413 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:36.798795 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:36.798799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:36.798801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:36.799093 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:36.799237 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:36.800155 (+ 918us) server_negotiation.cc:299] Negotiation successful
0923 17:39:36.800338 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:39:36.801834 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.782186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:36485 (local address 127.20.240.67:56187)
0923 17:39:36.782904 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:36.782948 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:36.784657 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:36.785363 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:36.785371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:36.785884 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:36.786720 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:36.786754 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.788072 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.788077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:36.789777 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:36.789790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.790698 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.790703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:36.790922 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.791616 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:39:36.791639 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:36.794086 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:36.797522 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:36.797531 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:36.797546 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:36.798144 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:36.798499 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:36.798502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:36.798505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:36.798706 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:36.799228 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:36.799235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:36.799712 (+ 477us) client_negotiation.cc:769] Sending connection context
0923 17:39:36.800027 (+ 315us) client_negotiation.cc:240] Negotiation successful
0923 17:39:36.800513 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":274,"threads_started":1}
I20260923 17:39:36.803380 7723 heartbeater.cc:347] Connected to a master server at 127.20.240.126:36485
I20260923 17:39:36.803668 7723 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:36.804435 7723 heartbeater.cc:511] Master 127.20.240.126:36485 requested a full tablet report, sending...
I20260923 17:39:36.805932 7266 ts_manager.cc:194] Registered new tserver with Master: 095b882d941b4cb0925e535a7e44c7d3 (127.20.240.67:43943)
I20260923 17:39:36.806759 7266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:56187
I20260923 17:39:36.815516 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:39:36.833660 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.820075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36016 (local address 127.20.240.126:36485)
0923 17:39:36.820322 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:36.820330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:36.820724 (+ 394us) server_negotiation.cc:407] Connection header received
0923 17:39:36.820869 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:36.820874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:36.820952 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:36.821037 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:36.822560 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.823487 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.824953 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.825375 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.825480 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.828258 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:36.828263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:36.828266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:36.828306 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:36.830640 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:36.831519 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:36.831523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:36.831525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:36.831678 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:36.832141 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:36.832144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:36.832147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:36.832481 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:36.832753 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:36.833158 (+ 405us) server_negotiation.cc:299] Negotiation successful
0923 17:39:36.833354 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:39:36.838080 7266 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:39:36.845301 7266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:39:36.871528 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.858524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:37401 (local address 127.0.0.1:59566)
0923 17:39:36.859565 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:36.859611 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:36.859805 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:36.862107 (+ 2302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:36.862116 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:36.862163 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:36.862664 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:36.862679 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.865166 (+ 2487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.865173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:36.867939 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:36.867965 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.870306 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.870314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:36.870591 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.870669 (+ 78us) client_negotiation.cc:769] Sending connection context
0923 17:39:36.870807 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 17:39:36.871060 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"threads_started":1}
I20260923 17:39:36.875185 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.858855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59566 (local address 127.20.240.65:37401)
0923 17:39:36.861361 (+ 2506us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:36.861372 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:36.861404 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:39:36.861518 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:36.861527 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:36.861817 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:36.862898 (+ 1081us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:36.863223 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.864967 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.868250 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.869999 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.874142 (+ 4143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.874487 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:36.874589 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 17:39:36.874770 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2308,"thread_start_us":2305,"threads_started":1}
I20260923 17:39:36.879467 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.858756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:45643 (local address 127.0.0.1:51772)
0923 17:39:36.871835 (+ 13079us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:36.871859 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:36.872004 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:36.873476 (+ 1472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:36.873480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:36.873506 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:36.873690 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:36.873701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.875630 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.875635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:36.877356 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:36.877374 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.878654 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.878660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:36.878844 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.878878 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:39:36.878966 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 17:39:36.879120 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12938,"thread_start_us":720,"threads_started":1}
I20260923 17:39:36.879524 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.859327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51772 (local address 127.20.240.66:45643)
0923 17:39:36.860407 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:36.860417 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:36.872280 (+ 11863us) server_negotiation.cc:407] Connection header received
0923 17:39:36.872410 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:36.872419 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:36.872730 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:36.872950 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:36.873870 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.875461 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.877521 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.878469 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.878623 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.878848 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:36.879027 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 17:39:36.879247 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":903,"thread_start_us":1334,"threads_started":1}
I20260923 17:39:36.879989 7390 tablet_service.cc:1467] Processing CreateTablet for tablet 4214f6ac9dfd4fe2977e9537bdb14fe1 (DEFAULT_TABLE table=test-table [id=99e25be1080e4d859cd3ea561b9436c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:36.881804 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.860896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38862 (local address 127.20.240.67:43943)
0923 17:39:36.871887 (+ 10991us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:36.871899 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:36.872778 (+ 879us) server_negotiation.cc:407] Connection header received
0923 17:39:36.873102 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:36.873112 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:36.873378 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:36.873560 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:36.874716 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.876231 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.878911 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.880325 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.880425 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.880650 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:36.881280 (+ 630us) server_negotiation.cc:299] Negotiation successful
0923 17:39:36.881464 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10768,"thread_start_us":11011,"threads_started":1}
I20260923 17:39:36.881852 7390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4214f6ac9dfd4fe2977e9537bdb14fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:36.882601 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.860392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:43943 (local address 127.0.0.1:38862)
0923 17:39:36.872640 (+ 12248us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:36.872661 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:36.872907 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:36.873912 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:36.873918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:36.873946 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:36.874463 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:36.874476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.876411 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.876417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:36.878688 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:36.878704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.880864 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.880871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:36.881062 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.881097 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:39:36.882022 (+ 925us) client_negotiation.cc:240] Negotiation successful
0923 17:39:36.882237 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12083,"thread_start_us":14021,"threads_started":1}
I20260923 17:39:36.884562 7524 tablet_service.cc:1467] Processing CreateTablet for tablet 4214f6ac9dfd4fe2977e9537bdb14fe1 (DEFAULT_TABLE table=test-table [id=99e25be1080e4d859cd3ea561b9436c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:36.886265 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4214f6ac9dfd4fe2977e9537bdb14fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:36.894390 7657 tablet_service.cc:1467] Processing CreateTablet for tablet 4214f6ac9dfd4fe2977e9537bdb14fe1 (DEFAULT_TABLE table=test-table [id=99e25be1080e4d859cd3ea561b9436c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:36.895761 7657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4214f6ac9dfd4fe2977e9537bdb14fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:36.901124 7741 tablet_bootstrap.cc:492] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3: Bootstrap starting.
I20260923 17:39:36.903460 7742 tablet_bootstrap.cc:492] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c: Bootstrap starting.
I20260923 17:39:36.905803 7741 tablet_bootstrap.cc:654] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:36.906917 7742 tablet_bootstrap.cc:654] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:36.907109 7741 log.cc:824] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:36.908130 7742 log.cc:824] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:36.910585 7745 tablet_bootstrap.cc:492] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3: Bootstrap starting.
I20260923 17:39:36.910573 7741 tablet_bootstrap.cc:492] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3: No bootstrap required, opened a new log
I20260923 17:39:36.910988 7742 tablet_bootstrap.cc:492] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c: No bootstrap required, opened a new log
I20260923 17:39:36.911301 7742 ts_tablet_manager.cc:1397] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260923 17:39:36.911334 7741 ts_tablet_manager.cc:1397] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260923 17:39:36.914167 7745 tablet_bootstrap.cc:654] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:36.915549 7745 log.cc:824] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:36.919382 7745 tablet_bootstrap.cc:492] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3: No bootstrap required, opened a new log
I20260923 17:39:36.919701 7745 ts_tablet_manager.cc:1397] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260923 17:39:36.919675 7742 raft_consensus.cc:348] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a4acdb0e614082a963ab4160f04fa3" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37401 } } peers { permanent_uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45643 } } peers { permanent_uuid: "095b882d941b4cb0925e535a7e44c7d3" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43943 } }
I20260923 17:39:36.920099 7742 raft_consensus.cc:374] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:36.920200 7742 raft_consensus.cc:729] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2130a9a8ed4b3a8d1524dab61bf10c, State: Initialized, Role: FOLLOWER
I20260923 17:39:36.920827 7742 consensus_queue.cc:260] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a4acdb0e614082a963ab4160f04fa3" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37401 } } peers { permanent_uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45643 } } peers { permanent_uuid: "095b882d941b4cb0925e535a7e44c7d3" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43943 } }
I20260923 17:39:36.921684 7741 raft_consensus.cc:348] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a4acdb0e614082a963ab4160f04fa3" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37401 } } peers { permanent_uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45643 } } peers { permanent_uuid: "095b882d941b4cb0925e535a7e44c7d3" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43943 } }
I20260923 17:39:36.922029 7589 heartbeater.cc:503] Master 127.20.240.126:36485 was elected leader, sending a full tablet report...
I20260923 17:39:36.922127 7741 raft_consensus.cc:374] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:36.922227 7741 raft_consensus.cc:729] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a4acdb0e614082a963ab4160f04fa3, State: Initialized, Role: FOLLOWER
I20260923 17:39:36.922828 7741 consensus_queue.cc:260] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a4acdb0e614082a963ab4160f04fa3" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37401 } } peers { permanent_uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45643 } } peers { permanent_uuid: "095b882d941b4cb0925e535a7e44c7d3" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43943 } }
I20260923 17:39:36.927937 7742 ts_tablet_manager.cc:1428] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c: Time spent starting tablet: real 0.017s user 0.010s sys 0.001s
I20260923 17:39:36.931100 7741 ts_tablet_manager.cc:1428] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3: Time spent starting tablet: real 0.020s user 0.004s sys 0.013s
I20260923 17:39:36.931367 7745 raft_consensus.cc:348] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a4acdb0e614082a963ab4160f04fa3" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37401 } } peers { permanent_uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45643 } } peers { permanent_uuid: "095b882d941b4cb0925e535a7e44c7d3" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43943 } }
I20260923 17:39:36.931381 7749 raft_consensus.cc:482] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:39:36.931847 7745 raft_consensus.cc:374] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:36.931959 7745 raft_consensus.cc:729] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095b882d941b4cb0925e535a7e44c7d3, State: Initialized, Role: FOLLOWER
I20260923 17:39:36.932049 7749 raft_consensus.cc:504] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a4acdb0e614082a963ab4160f04fa3" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37401 } } peers { permanent_uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45643 } } peers { permanent_uuid: "095b882d941b4cb0925e535a7e44c7d3" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43943 } }
I20260923 17:39:36.932566 7745 consensus_queue.cc:260] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a4acdb0e614082a963ab4160f04fa3" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37401 } } peers { permanent_uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45643 } } peers { permanent_uuid: "095b882d941b4cb0925e535a7e44c7d3" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43943 } }
I20260923 17:39:36.933564 7723 heartbeater.cc:503] Master 127.20.240.126:36485 was elected leader, sending a full tablet report...
I20260923 17:39:36.934072 7749 leader_election.cc:288] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c2130a9a8ed4b3a8d1524dab61bf10c (127.20.240.66:45643), 095b882d941b4cb0925e535a7e44c7d3 (127.20.240.67:43943)
I20260923 17:39:36.936107 7745 ts_tablet_manager.cc:1428] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3: Time spent starting tablet: real 0.016s user 0.013s sys 0.001s
I20260923 17:39:36.943976 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.934841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:38839 (local address 127.20.240.66:45643)
0923 17:39:36.935040 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:36.935046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:36.937153 (+ 2107us) server_negotiation.cc:407] Connection header received
0923 17:39:36.937409 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:36.937416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:36.937501 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:36.937571 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:36.938490 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.939416 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.941920 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.942946 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.943077 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.943134 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:36.943644 (+ 510us) server_negotiation.cc:299] Negotiation successful
0923 17:39:36.943772 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:39:36.944195 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.934346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:45643 (local address 127.20.240.65:38839)
0923 17:39:36.937043 (+ 2697us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:36.937065 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:36.937222 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:36.937793 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:36.937796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:36.937837 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:36.938275 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:36.938291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.939615 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.939620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:36.941753 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:36.941767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.943226 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.943234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:36.943457 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.943505 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 17:39:36.943622 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 17:39:36.943773 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2430,"thread_start_us":2231,"threads_started":1}
I20260923 17:39:36.945345 7544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4214f6ac9dfd4fe2977e9537bdb14fe1" candidate_uuid: "05a4acdb0e614082a963ab4160f04fa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c" is_pre_election: true
I20260923 17:39:36.945919 7544 raft_consensus.cc:2454] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05a4acdb0e614082a963ab4160f04fa3 in term 0.
I20260923 17:39:36.946609 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.937515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:44749 (local address 127.20.240.67:43943)
0923 17:39:36.937716 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:36.937722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:36.940342 (+ 2620us) server_negotiation.cc:407] Connection header received
0923 17:39:36.940647 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:36.940652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:36.940711 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:36.940782 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:36.941445 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.942379 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.944699 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.945516 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.945700 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.945767 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:36.946185 (+ 418us) server_negotiation.cc:299] Negotiation successful
0923 17:39:36.946300 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:39:36.946627 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.937054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:43943 (local address 127.20.240.65:44749)
0923 17:39:36.940234 (+ 3180us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:36.940254 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:36.940510 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:36.940823 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:36.940828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:36.940855 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:36.941280 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:36.941290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.942542 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.942546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:36.944542 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:36.944558 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.945780 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.945788 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:36.945981 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.946019 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 17:39:36.946129 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 17:39:36.946270 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3068,"thread_start_us":2896,"threads_started":1}
I20260923 17:39:36.946763 7345 leader_election.cc:302] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a4acdb0e614082a963ab4160f04fa3, 0c2130a9a8ed4b3a8d1524dab61bf10c; no voters:
I20260923 17:39:36.947261 7749 raft_consensus.cc:2781] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:39:36.947435 7749 raft_consensus.cc:482] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:39:36.947571 7749 raft_consensus.cc:3037] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:36.948029 7678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4214f6ac9dfd4fe2977e9537bdb14fe1" candidate_uuid: "05a4acdb0e614082a963ab4160f04fa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "095b882d941b4cb0925e535a7e44c7d3" is_pre_election: true
I20260923 17:39:36.948609 7678 raft_consensus.cc:2454] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05a4acdb0e614082a963ab4160f04fa3 in term 0.
I20260923 17:39:36.950519 7749 raft_consensus.cc:504] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a4acdb0e614082a963ab4160f04fa3" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37401 } } peers { permanent_uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45643 } } peers { permanent_uuid: "095b882d941b4cb0925e535a7e44c7d3" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43943 } }
I20260923 17:39:36.951248 7749 leader_election.cc:288] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [CANDIDATE]: Term 1 election: Requested vote from peers 0c2130a9a8ed4b3a8d1524dab61bf10c (127.20.240.66:45643), 095b882d941b4cb0925e535a7e44c7d3 (127.20.240.67:43943)
I20260923 17:39:36.951749 7544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4214f6ac9dfd4fe2977e9537bdb14fe1" candidate_uuid: "05a4acdb0e614082a963ab4160f04fa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c"
I20260923 17:39:36.951745 7678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4214f6ac9dfd4fe2977e9537bdb14fe1" candidate_uuid: "05a4acdb0e614082a963ab4160f04fa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "095b882d941b4cb0925e535a7e44c7d3"
I20260923 17:39:36.951956 7544 raft_consensus.cc:3037] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:36.951952 7678 raft_consensus.cc:3037] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:36.954257 7544 raft_consensus.cc:2454] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05a4acdb0e614082a963ab4160f04fa3 in term 1.
I20260923 17:39:36.954346 7678 raft_consensus.cc:2454] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05a4acdb0e614082a963ab4160f04fa3 in term 1.
I20260923 17:39:36.954717 7345 leader_election.cc:302] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a4acdb0e614082a963ab4160f04fa3, 0c2130a9a8ed4b3a8d1524dab61bf10c; no voters:
I20260923 17:39:36.955057 7749 raft_consensus.cc:2781] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:36.955276 7749 raft_consensus.cc:686] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [term 1 LEADER]: Becoming Leader. State: Replica: 05a4acdb0e614082a963ab4160f04fa3, State: Running, Role: LEADER
I20260923 17:39:36.955770 7749 consensus_queue.cc:237] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a4acdb0e614082a963ab4160f04fa3" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37401 } } peers { permanent_uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45643 } } peers { permanent_uuid: "095b882d941b4cb0925e535a7e44c7d3" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43943 } }
I20260923 17:39:36.960456 7266 catalog_manager.cc:5495] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05a4acdb0e614082a963ab4160f04fa3 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "05a4acdb0e614082a963ab4160f04fa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05a4acdb0e614082a963ab4160f04fa3" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "095b882d941b4cb0925e535a7e44c7d3" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43943 } health_report { overall_health: UNKNOWN } } }
I20260923 17:39:36.978317 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:36.972793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59578 (local address 127.20.240.65:37401)
0923 17:39:36.973013 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:36.973019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:36.973039 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:39:36.973211 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:36.973215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:36.973294 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:36.973451 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:39:36.973833 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.974564 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.976047 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:36.976469 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:36.976589 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:36.977190 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:39:36.977708 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:39:36.977880 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:36.978041 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 17:39:36.978090 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:39:36.987500 7678 raft_consensus.cc:1261] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3 [term 1 FOLLOWER]: Refusing update from remote peer 05a4acdb0e614082a963ab4160f04fa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:39:36.987512 7544 raft_consensus.cc:1261] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c [term 1 FOLLOWER]: Refusing update from remote peer 05a4acdb0e614082a963ab4160f04fa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:39:36.988662 7748 consensus_queue.cc:1035] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "095b882d941b4cb0925e535a7e44c7d3" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:39:36.989050 7749 consensus_queue.cc:1035] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c2130a9a8ed4b3a8d1524dab61bf10c" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:39:37.017518 7265 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:39:37.017899 7265 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:39:37.021842 7265 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e629591aef044ceeb04c7e8feb4af58b: Sending DeleteTablet for 3 replicas of tablet 4214f6ac9dfd4fe2977e9537bdb14fe1
I20260923 17:39:37.023131 7390 tablet_service.cc:1514] Processing DeleteTablet for tablet 4214f6ac9dfd4fe2977e9537bdb14fe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:37 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:59566
I20260923 17:39:37.023219 7657 tablet_service.cc:1514] Processing DeleteTablet for tablet 4214f6ac9dfd4fe2977e9537bdb14fe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:37 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:38862
I20260923 17:39:37.023452 7524 tablet_service.cc:1514] Processing DeleteTablet for tablet 4214f6ac9dfd4fe2977e9537bdb14fe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:37 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:51772
I20260923 17:39:37.028952 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 7327
I20260923 17:39:37.034161 7766 tablet_replica.cc:330] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3: stopping tablet replica
I20260923 17:39:37.034878 7766 raft_consensus.cc:2229] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:39:37.035804 7766 raft_consensus.cc:2258] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:37.036690 7767 tablet_replica.cc:330] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3: stopping tablet replica
I20260923 17:39:37.036931 7768 tablet_replica.cc:330] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c: stopping tablet replica
I20260923 17:39:37.037245 7767 raft_consensus.cc:2229] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:39:37.037390 7768 raft_consensus.cc:2229] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:39:37.037743 7767 raft_consensus.cc:2258] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:37.037861 7768 raft_consensus.cc:2258] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:37.038779 7766 ts_tablet_manager.cc:1905] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:37.039731 7767 ts_tablet_manager.cc:1905] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:37.040202 7768 ts_tablet_manager.cc:1905] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:37.046244 7766 ts_tablet_manager.cc:1918] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:37.046293 7767 ts_tablet_manager.cc:1918] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:37.046448 7767 log.cc:1196] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/4214f6ac9dfd4fe2977e9537bdb14fe1
I20260923 17:39:37.046454 7766 log.cc:1196] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/4214f6ac9dfd4fe2977e9537bdb14fe1
I20260923 17:39:37.046962 7767 ts_tablet_manager.cc:1939] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 095b882d941b4cb0925e535a7e44c7d3: Deleting consensus metadata
I20260923 17:39:37.046973 7766 ts_tablet_manager.cc:1939] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 05a4acdb0e614082a963ab4160f04fa3: Deleting consensus metadata
I20260923 17:39:37.047657 7768 ts_tablet_manager.cc:1918] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:37.047945 7768 log.cc:1196] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/4214f6ac9dfd4fe2977e9537bdb14fe1
I20260923 17:39:37.048540 7768 ts_tablet_manager.cc:1939] T 4214f6ac9dfd4fe2977e9537bdb14fe1 P 0c2130a9a8ed4b3a8d1524dab61bf10c: Deleting consensus metadata
I20260923 17:39:37.048882 7250 catalog_manager.cc:4841] TS 05a4acdb0e614082a963ab4160f04fa3 (127.20.240.65:37401): tablet 4214f6ac9dfd4fe2977e9537bdb14fe1 (table test-table [id=99e25be1080e4d859cd3ea561b9436c6]) successfully deleted
I20260923 17:39:37.049211 7252 catalog_manager.cc:4841] TS 095b882d941b4cb0925e535a7e44c7d3 (127.20.240.67:43943): tablet 4214f6ac9dfd4fe2977e9537bdb14fe1 (table test-table [id=99e25be1080e4d859cd3ea561b9436c6]) successfully deleted
I20260923 17:39:37.050336 7252 catalog_manager.cc:4841] TS 0c2130a9a8ed4b3a8d1524dab61bf10c (127.20.240.66:45643): tablet 4214f6ac9dfd4fe2977e9537bdb14fe1 (table test-table [id=99e25be1080e4d859cd3ea561b9436c6]) successfully deleted
Sep 23 17:39:37 dist-test-slave-ppjl krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:39:37.065946 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:37.030452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.20.240.65:37401)
0923 17:39:37.030654 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:37.030661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:37.049962 (+ 19301us) server_negotiation.cc:407] Connection header received
0923 17:39:37.050206 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:37.050212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:37.050297 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:37.050412 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:37.051791 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:37.053013 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:37.054447 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:37.055199 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:37.055340 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:37.059836 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:37.059851 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:37.059867 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:37.059907 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:37.062947 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:37.063776 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:37.063784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:37.063786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:37.063955 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:37.064358 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:37.064362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:37.064364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:37.064735 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:37.065065 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:37.065342 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 17:39:37.065594 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:39:37.066598 7450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:37.166213 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 7327
I20260923 17:39:37.184707 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 7461
Sep 23 17:39:37 dist-test-slave-ppjl krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:39:37.200330 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:37.185658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51788 (local address 127.20.240.66:45643)
0923 17:39:37.185813 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:37.185818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:37.185836 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:39:37.186011 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:37.186015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:37.186088 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:37.186171 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:37.187428 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:37.188290 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:37.189570 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:37.189976 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:37.190076 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:37.194259 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:37.194277 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:37.194295 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:37.194340 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:37.197375 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:37.198087 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:37.198095 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:37.198097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:37.198247 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:37.198622 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:37.198625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:37.198627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:37.199106 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:37.199410 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:37.199634 (+ 224us) server_negotiation.cc:299] Negotiation successful
0923 17:39:37.199808 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:39:37.201189 7584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:37.291002 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 7461
I20260923 17:39:37.305189 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 7594
Sep 23 17:39:37 dist-test-slave-ppjl krb5kdc[7221](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:39:37.319900 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:37.305968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.20.240.67:43943)
0923 17:39:37.306171 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:37.306175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:37.306195 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:39:37.306303 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:37.306307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:37.306382 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:37.306462 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:37.307864 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:37.308642 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:37.310007 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:37.310421 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:37.310517 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:37.314594 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:37.314608 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:37.314624 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:37.314666 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:37.317330 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:37.317951 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:37.317959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:37.317961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:37.318133 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:37.318460 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:37.318463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:37.318465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:37.318824 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:37.319029 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:37.319419 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 17:39:37.319594 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:39:37.320583 7718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:37.409480 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 7594
I20260923 17:39:37.423059 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 7234
I20260923 17:39:37.424238 7296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:37.536204 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 7234
2026-09-23T17:39:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4097 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T17:39:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:39:37Z Disabled control of system clock
I20260923 17:39:37.607913 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41501
--webserver_interface=127.20.240.126
--webserver_port=0
--builtin_ntp_servers=127.20.240.84:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:41501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:39:37.943027 7777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:37.943356 7777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:37.943472 7777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:37.951023 7777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:39:37.951144 7777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:37.951202 7777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:37.951272 7777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:39:37.951336 7777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:39:37.960866 7777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:41501
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:37.962219 7777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:37.964098 7777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:37.975978 7782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:37.976035 7785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:37.976004 7783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:37.977180 7777 server_base.cc:1034] running on GCE node
I20260923 17:39:37.978065 7777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:37.979632 7777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:37.980840 7777 hybrid_clock.cc:648] HybridClock initialized: now 1790185177980785 us; error 44 us; skew 500 ppm
I20260923 17:39:37.981200 7777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:39:37.983487 7777 webserver.cc:460] Webserver started at http://127.20.240.126:43325/ using document root <none> and password file <none>
I20260923 17:39:37.984174 7777 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:37.984273 7777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:37.984587 7777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:37.987610 7777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "c57762f419dd42dca175d93cac8d2be7"
format_stamp: "Formatted at 2026-09-23 17:39:37 on dist-test-slave-ppjl"
server_key: "e8adf3be33def5e4b2d5211ef222462e"
server_key_iv: "8b1265c8dad76970bf9ac0f40fe18b86"
server_key_version: "encryptionkey@0"
I20260923 17:39:37.988454 7777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "c57762f419dd42dca175d93cac8d2be7"
format_stamp: "Formatted at 2026-09-23 17:39:37 on dist-test-slave-ppjl"
server_key: "e8adf3be33def5e4b2d5211ef222462e"
server_key_iv: "8b1265c8dad76970bf9ac0f40fe18b86"
server_key_version: "encryptionkey@0"
I20260923 17:39:37.993734 7777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:39:37.997277 7791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:37.998735 7777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:39:37.998992 7777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "c57762f419dd42dca175d93cac8d2be7"
format_stamp: "Formatted at 2026-09-23 17:39:37 on dist-test-slave-ppjl"
server_key: "e8adf3be33def5e4b2d5211ef222462e"
server_key_iv: "8b1265c8dad76970bf9ac0f40fe18b86"
server_key_version: "encryptionkey@0"
I20260923 17:39:37.999235 7777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:38.017035 7777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:38.017915 7777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:38.018213 7777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:38.035806 7843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:41501 every 8 connection(s)
I20260923 17:39:38.035800 7777 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:41501
I20260923 17:39:38.037393 7777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:39:38.041334 7844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:38.042035 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 7777
I20260923 17:39:38.042215 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:39:38.042658 21441 external_mini_cluster.cc:1428] Setting key c287d99419f4dfce98ff0b34d8086c04
I20260923 17:39:38.056597 7844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7: Bootstrap starting.
I20260923 17:39:38.058781 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:38.044618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47720 (local address 127.20.240.126:41501)
0923 17:39:38.045713 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:38.045728 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:38.045783 (+ 55us) server_negotiation.cc:407] Connection header received
0923 17:39:38.048013 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:38.048046 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:38.048744 (+ 698us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:38.049217 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:38.049776 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:38.051688 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:38.053210 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:38.054340 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:38.054551 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:38.055691 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:38.055702 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:38.055723 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:38.056459 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:38.056569 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:38.056602 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:38.056736 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:38.057042 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 17:39:38.057407 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":310,"threads_started":1}
I20260923 17:39:38.060499 7844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:38.061590 7844 log.cc:824] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:38.064733 7844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7: No bootstrap required, opened a new log
I20260923 17:39:38.072568 7844 raft_consensus.cc:348] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57762f419dd42dca175d93cac8d2be7" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41501 } }
I20260923 17:39:38.073073 7844 raft_consensus.cc:374] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:38.073184 7844 raft_consensus.cc:729] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c57762f419dd42dca175d93cac8d2be7, State: Initialized, Role: FOLLOWER
I20260923 17:39:38.073799 7844 consensus_queue.cc:260] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57762f419dd42dca175d93cac8d2be7" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41501 } }
I20260923 17:39:38.074031 7844 raft_consensus.cc:388] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:39:38.074162 7844 raft_consensus.cc:482] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:39:38.074309 7844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:38.076603 7844 raft_consensus.cc:504] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57762f419dd42dca175d93cac8d2be7" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41501 } }
I20260923 17:39:38.077147 7844 leader_election.cc:302] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c57762f419dd42dca175d93cac8d2be7; no voters:
I20260923 17:39:38.077800 7844 leader_election.cc:288] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:39:38.077925 7849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:38.080018 7844 sys_catalog.cc:564] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:39:38.081943 7849 raft_consensus.cc:686] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [term 1 LEADER]: Becoming Leader. State: Replica: c57762f419dd42dca175d93cac8d2be7, State: Running, Role: LEADER
I20260923 17:39:38.082508 7849 consensus_queue.cc:237] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57762f419dd42dca175d93cac8d2be7" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41501 } }
I20260923 17:39:38.085513 7851 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c57762f419dd42dca175d93cac8d2be7. Latest consensus state: current_term: 1 leader_uuid: "c57762f419dd42dca175d93cac8d2be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57762f419dd42dca175d93cac8d2be7" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41501 } } }
I20260923 17:39:38.085758 7850 sys_catalog.cc:455] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c57762f419dd42dca175d93cac8d2be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c57762f419dd42dca175d93cac8d2be7" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41501 } } }
I20260923 17:39:38.085945 7851 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:38.086014 7850 sys_catalog.cc:458] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7 [sys.catalog]: This master's current role is: LEADER
W20260923 17:39:38.090894 7863 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:39:38.091006 7863 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:39:38.091106 7864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:39:38.092576 7864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:39:38.101231 7864 catalog_manager.cc:1347] Generated new cluster ID: b834d9652e744c598caffbf8b4b1255f
I20260923 17:39:38.101356 7864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:39:38.112260 7864 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:39:38.113559 7864 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:39:38.119598 7864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7: Generated new TSK 0
I20260923 17:39:38.120507 7864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:39:38.133484 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41501
--builtin_ntp_servers=127.20.240.84:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:39:38.485419 7868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:38.485713 7868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:38.485845 7868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:38.493316 7868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:38.493427 7868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:38.493563 7868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:39:38.502964 7868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:38.504350 7868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:38.506165 7868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:38.519258 7873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:38.519228 7874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:38.519228 7876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:38.520004 7868 server_base.cc:1034] running on GCE node
I20260923 17:39:38.520685 7868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:38.521734 7868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:38.522966 7868 hybrid_clock.cc:648] HybridClock initialized: now 1790185178522914 us; error 45 us; skew 500 ppm
I20260923 17:39:38.523342 7868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:39:38.525621 7868 webserver.cc:460] Webserver started at http://127.20.240.65:33675/ using document root <none> and password file <none>
I20260923 17:39:38.526340 7868 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:38.526439 7868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:38.526747 7868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:38.530048 7868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "9349631a01314397b09e1b7dafb2062b"
format_stamp: "Formatted at 2026-09-23 17:39:38 on dist-test-slave-ppjl"
server_key: "6ecff493cf7342de2c108828648ccf6c"
server_key_iv: "79e4d657e6447763d50b9818189b1d5b"
server_key_version: "encryptionkey@0"
I20260923 17:39:38.530876 7868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "9349631a01314397b09e1b7dafb2062b"
format_stamp: "Formatted at 2026-09-23 17:39:38 on dist-test-slave-ppjl"
server_key: "6ecff493cf7342de2c108828648ccf6c"
server_key_iv: "79e4d657e6447763d50b9818189b1d5b"
server_key_version: "encryptionkey@0"
I20260923 17:39:38.536743 7868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:39:38.540663 7882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:38.542135 7868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:39:38.542294 7868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "9349631a01314397b09e1b7dafb2062b"
format_stamp: "Formatted at 2026-09-23 17:39:38 on dist-test-slave-ppjl"
server_key: "6ecff493cf7342de2c108828648ccf6c"
server_key_iv: "79e4d657e6447763d50b9818189b1d5b"
server_key_version: "encryptionkey@0"
I20260923 17:39:38.542510 7868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:38.586887 7868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:38.587878 7868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:38.588217 7868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:38.589304 7868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:38.590935 7868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:38.591051 7868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:38.591212 7868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:38.591300 7868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:38.624383 7868 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:34707
I20260923 17:39:38.624416 7995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:34707 every 8 connection(s)
I20260923 17:39:38.625672 7868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:39:38.634788 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 7868
I20260923 17:39:38.634958 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:39:38.635433 21441 external_mini_cluster.cc:1428] Setting key 44e5deb9e55968f4063aa2024ea6e546
I20260923 17:39:38.638331 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41501
--builtin_ntp_servers=127.20.240.84:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:39:38.639735 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:38.628274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:60775 (local address 127.20.240.126:41501)
0923 17:39:38.628744 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:38.628748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:38.629846 (+ 1098us) server_negotiation.cc:407] Connection header received
0923 17:39:38.632046 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:38.632052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:38.632140 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:38.632237 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:38.633414 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:38.634134 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:38.635833 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:38.636306 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:38.636434 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:38.638137 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:38.638142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:38.638145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:38.638162 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:38.638206 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:38.638218 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:38.638294 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:38.639272 (+ 978us) server_negotiation.cc:299] Negotiation successful
0923 17:39:38.639394 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":258,"threads_started":1}
I20260923 17:39:38.640244 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:38.628791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41501 (local address 127.20.240.65:60775)
0923 17:39:38.629686 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:38.629728 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:38.631733 (+ 2005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:38.632504 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:38.632514 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:38.632627 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:38.633230 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:38.633249 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:38.634289 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:38.634293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:38.635685 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:38.635698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:38.636566 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:38.636573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:38.636882 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:38.637766 (+ 884us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:39:38.637771 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:38.637835 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:39:38.637947 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:38.638307 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:38.638319 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:38.638328 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:39:38.638495 (+ 167us) client_negotiation.cc:240] Negotiation successful
0923 17:39:38.638864 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":266,"threads_started":1}
I20260923 17:39:38.641948 7996 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41501
I20260923 17:39:38.642251 7996 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:38.643023 7996 heartbeater.cc:511] Master 127.20.240.126:41501 requested a full tablet report, sending...
I20260923 17:39:38.645303 7808 ts_manager.cc:194] Registered new tserver with Master: 9349631a01314397b09e1b7dafb2062b (127.20.240.65:34707)
I20260923 17:39:38.647080 7808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.65:60775
W20260923 17:39:38.971360 8000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:38.971662 8000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:38.971774 8000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:38.979233 8000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:38.979354 8000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:38.979503 8000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:39:38.989125 8000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:38.990499 8000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:38.992379 8000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:39.005870 8006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:39.005941 8008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:39.006606 8005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:39.006685 8000 server_base.cc:1034] running on GCE node
I20260923 17:39:39.007596 8000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:39.008908 8000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:39.010124 8000 hybrid_clock.cc:648] HybridClock initialized: now 1790185179010050 us; error 71 us; skew 500 ppm
I20260923 17:39:39.010483 8000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:39:39.012923 8000 webserver.cc:460] Webserver started at http://127.20.240.66:34495/ using document root <none> and password file <none>
I20260923 17:39:39.013667 8000 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:39.013773 8000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:39.014098 8000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:39.017174 8000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "3177e6b3fcd643cb96d0a9b733f6719d"
format_stamp: "Formatted at 2026-09-23 17:39:39 on dist-test-slave-ppjl"
server_key: "165fcc7dc05a1e246d9155d023548a9d"
server_key_iv: "564dfd453eaec2eded4ff795e622b6a4"
server_key_version: "encryptionkey@0"
I20260923 17:39:39.018018 8000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "3177e6b3fcd643cb96d0a9b733f6719d"
format_stamp: "Formatted at 2026-09-23 17:39:39 on dist-test-slave-ppjl"
server_key: "165fcc7dc05a1e246d9155d023548a9d"
server_key_iv: "564dfd453eaec2eded4ff795e622b6a4"
server_key_version: "encryptionkey@0"
I20260923 17:39:39.023869 8000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:39:39.027781 8014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:39.029347 8000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:39:39.029557 8000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "3177e6b3fcd643cb96d0a9b733f6719d"
format_stamp: "Formatted at 2026-09-23 17:39:39 on dist-test-slave-ppjl"
server_key: "165fcc7dc05a1e246d9155d023548a9d"
server_key_iv: "564dfd453eaec2eded4ff795e622b6a4"
server_key_version: "encryptionkey@0"
I20260923 17:39:39.029932 8000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:39.070990 8000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:39.071918 8000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:39.072218 8000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:39.073267 8000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:39.074842 8000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:39.074941 8000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:39.075047 8000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:39.075112 8000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:39.105993 8000 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:42797
I20260923 17:39:39.106060 8127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:42797 every 8 connection(s)
I20260923 17:39:39.107314 8000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:39:39.115698 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 8000
I20260923 17:39:39.115842 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:39:39.116282 21441 external_mini_cluster.cc:1428] Setting key 3c75e657ea70340e47bb7ffa097ea0b7
I20260923 17:39:39.119022 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41501
--builtin_ntp_servers=127.20.240.84:32961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:39:39.120618 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.109911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:60131 (local address 127.20.240.126:41501)
0923 17:39:39.110098 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:39.110103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:39.111648 (+ 1545us) server_negotiation.cc:407] Connection header received
0923 17:39:39.113581 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:39.113588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:39.113678 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:39.113795 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:39.114972 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.115851 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.117590 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.118041 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.118169 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.119833 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:39.119840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:39.119853 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:39.119876 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:39.119917 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:39.119932 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:39.120018 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:39.120211 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 17:39:39.120315 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:39:39.121630 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.110436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41501 (local address 127.20.240.66:60131)
0923 17:39:39.111483 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:39.111527 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:39.113255 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:39.114067 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:39.114078 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:39.114172 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:39.114782 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:39.114803 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.116031 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.116035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:39.117428 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:39.117443 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.118290 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.118299 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:39.118550 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.119408 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:39:39.119411 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:39.119468 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:39:39.119609 (+ 141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:39.120032 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:39.120040 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:39.120045 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:39:39.120183 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 17:39:39.120503 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":368,"threads_started":1}
I20260923 17:39:39.123188 8128 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41501
I20260923 17:39:39.123494 8128 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:39.124265 8128 heartbeater.cc:511] Master 127.20.240.126:41501 requested a full tablet report, sending...
I20260923 17:39:39.125756 7808 ts_manager.cc:194] Registered new tserver with Master: 3177e6b3fcd643cb96d0a9b733f6719d (127.20.240.66:42797)
I20260923 17:39:39.126642 7808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.66:60131
W20260923 17:39:39.459792 8131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:39.460206 8131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:39.460350 8131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:39.468056 8131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:39.468189 8131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:39.468348 8131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:39:39.477759 8131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:32961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:39.479205 8131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:39.481647 8131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:39.497119 8139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:39.497725 8136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:39.497767 8137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:39.503247 8131 server_base.cc:1034] running on GCE node
I20260923 17:39:39.504074 8131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:39.505340 8131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:39.506579 8131 hybrid_clock.cc:648] HybridClock initialized: now 1790185179506512 us; error 69 us; skew 500 ppm
I20260923 17:39:39.506925 8131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:39:39.509508 8131 webserver.cc:460] Webserver started at http://127.20.240.67:41699/ using document root <none> and password file <none>
I20260923 17:39:39.510277 8131 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:39.510385 8131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:39.510704 8131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:39.513886 8131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "422c0e771d3e45b39f670f77506dc497"
format_stamp: "Formatted at 2026-09-23 17:39:39 on dist-test-slave-ppjl"
server_key: "659d8d110984c6822f71d23782e8cc6d"
server_key_iv: "fee5bb8e8a5e864ec1809a449bd992dd"
server_key_version: "encryptionkey@0"
I20260923 17:39:39.514688 8131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "422c0e771d3e45b39f670f77506dc497"
format_stamp: "Formatted at 2026-09-23 17:39:39 on dist-test-slave-ppjl"
server_key: "659d8d110984c6822f71d23782e8cc6d"
server_key_iv: "fee5bb8e8a5e864ec1809a449bd992dd"
server_key_version: "encryptionkey@0"
I20260923 17:39:39.520354 8131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:39:39.524283 8145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:39.525822 8131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:39:39.525986 8131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "422c0e771d3e45b39f670f77506dc497"
format_stamp: "Formatted at 2026-09-23 17:39:39 on dist-test-slave-ppjl"
server_key: "659d8d110984c6822f71d23782e8cc6d"
server_key_iv: "fee5bb8e8a5e864ec1809a449bd992dd"
server_key_version: "encryptionkey@0"
I20260923 17:39:39.526212 8131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:39.541531 8131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:39.542421 8131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:39.542735 8131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:39.543895 8131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:39.545634 8131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:39.545746 8131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:39.545946 8131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:39.546093 8131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:39.579979 8131 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:36837
I20260923 17:39:39.580035 8258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:36837 every 8 connection(s)
I20260923 17:39:39.581538 8131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:39:39.584890 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 8131
I20260923 17:39:39.585031 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:39:39.585470 21441 external_mini_cluster.cc:1428] Setting key 4fb7a73b23aeeca8055bf81da8c2e647
I20260923 17:39:39.593945 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.584295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:55779 (local address 127.20.240.126:41501)
0923 17:39:39.584513 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:39.584519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:39.586122 (+ 1603us) server_negotiation.cc:407] Connection header received
0923 17:39:39.587976 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:39.587981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:39.588084 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:39.588188 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:39.589257 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.589988 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.591571 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.591963 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.592058 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.593305 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:39.593310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:39.593313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:39.593330 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:39.593363 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:39.593374 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:39.593441 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:39.593626 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 17:39:39.593707 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:39:39.594942 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.584827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41501 (local address 127.20.240.67:55779)
0923 17:39:39.585931 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:39.585976 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:39.587727 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:39.588429 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:39.588437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:39.588529 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:39.589125 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:39.589142 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.590108 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.590112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:39.591451 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:39.591463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.592154 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.592159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:39.592358 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.593019 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:39:39.593022 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:39.593061 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:39:39.593137 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:39.593458 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:39.593466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:39.593472 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:39:39.593606 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 17:39:39.593905 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":346,"threads_started":1}
I20260923 17:39:39.596267 8259 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41501
I20260923 17:39:39.596541 8259 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:39.597230 8259 heartbeater.cc:511] Master 127.20.240.126:41501 requested a full tablet report, sending...
I20260923 17:39:39.598558 7808 ts_manager.cc:194] Registered new tserver with Master: 422c0e771d3e45b39f670f77506dc497 (127.20.240.67:36837)
I20260923 17:39:39.599478 7808 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.67:55779
I20260923 17:39:39.600701 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:39:39.610594 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.604876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47736 (local address 127.20.240.126:41501)
0923 17:39:39.605150 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:39.605156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:39.605462 (+ 306us) server_negotiation.cc:407] Connection header received
0923 17:39:39.605618 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:39.605621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:39.605694 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:39.605774 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:39.606228 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.606940 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.608339 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.608747 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.608842 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.609862 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:39.609868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:39.609871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:39.609888 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:39.609971 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:39.609984 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:39.610108 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:39.610234 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 17:39:39.610280 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 17:39:39.614468 7808 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:39:39.621197 7808 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:39:39.648871 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.633341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34707 (local address 127.0.0.1:43238)
0923 17:39:39.634708 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:39.634749 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:39.634915 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:39.641131 (+ 6216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:39.641140 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:39.641179 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:39.641770 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:39.641787 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.643590 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.643595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:39.645908 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:39.645924 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.647883 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.647889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:39.648075 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.648149 (+ 74us) client_negotiation.cc:769] Sending connection context
0923 17:39:39.648288 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 17:39:39.648495 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":40,"thread_start_us":964,"threads_started":1}
I20260923 17:39:39.648887 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.633525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.20.240.65:34707)
0923 17:39:39.640401 (+ 6876us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:39.640411 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:39.640445 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:39:39.640562 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:39.640571 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:39.640864 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:39.641078 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:39.641937 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.643459 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.646175 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.647706 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.647850 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.648157 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:39.648338 (+ 181us) server_negotiation.cc:299] Negotiation successful
0923 17:39:39.648515 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6702,"thread_start_us":6582,"threads_started":1}
I20260923 17:39:39.648967 8273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.634011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:42797 (local address 127.0.0.1:58000)
0923 17:39:39.639617 (+ 5606us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:39.639639 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:39.639886 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:39.641407 (+ 1521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:39.641411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:39.641438 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:39.642644 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:39.642657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.644244 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.644250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:39.646083 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:39.646097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.648262 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.648268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:39.648449 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.648479 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 17:39:39.648552 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 17:39:39.648666 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5421,"mutex_wait_us":24,"thread_start_us":5292,"threads_started":1}
I20260923 17:39:39.649896 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.634351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.20.240.66:42797)
0923 17:39:39.640724 (+ 6373us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:39.640733 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:39.640760 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:39:39.640848 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:39.640855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:39.641081 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:39.641269 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:39.642921 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.644124 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.646311 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.647750 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.647894 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.648136 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:39.649509 (+ 1373us) server_negotiation.cc:299] Negotiation successful
0923 17:39:39.649638 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6227,"thread_start_us":6109,"threads_started":1}
I20260923 17:39:39.650632 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.635741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.20.240.67:36837)
0923 17:39:39.636461 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:39.636470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:39.642034 (+ 5564us) server_negotiation.cc:407] Connection header received
0923 17:39:39.642342 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:39.642351 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:39.642590 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:39.642746 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:39.643454 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.644977 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.647524 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.648881 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.649386 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.649635 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:39.650114 (+ 479us) server_negotiation.cc:299] Negotiation successful
0923 17:39:39.650278 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":382,"threads_started":1}
I20260923 17:39:39.650733 7996 heartbeater.cc:503] Master 127.20.240.126:41501 was elected leader, sending a full tablet report...
I20260923 17:39:39.651767 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.635140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:36837 (local address 127.0.0.1:47956)
0923 17:39:39.641912 (+ 6772us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:39.641929 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:39.642134 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:39.642961 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:39.642965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:39.642989 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:39.643290 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:39.643308 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.645126 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.645132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:39.647335 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:39.647350 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.649750 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.649756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:39.649920 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.649950 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 17:39:39.650880 (+ 930us) client_negotiation.cc:240] Negotiation successful
0923 17:39:39.651025 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6648,"thread_start_us":6461,"threads_started":1}
I20260923 17:39:39.652969 7930 tablet_service.cc:1467] Processing CreateTablet for tablet 9ebb0b5d13874128a7628dbea53f7035 (DEFAULT_TABLE table=test-table [id=fa13f48673a141b8b4991d96ec417c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:39.653066 8062 tablet_service.cc:1467] Processing CreateTablet for tablet 9ebb0b5d13874128a7628dbea53f7035 (DEFAULT_TABLE table=test-table [id=fa13f48673a141b8b4991d96ec417c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:39.654219 8062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebb0b5d13874128a7628dbea53f7035. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:39.654304 7930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebb0b5d13874128a7628dbea53f7035. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:39.657614 8193 tablet_service.cc:1467] Processing CreateTablet for tablet 9ebb0b5d13874128a7628dbea53f7035 (DEFAULT_TABLE table=test-table [id=fa13f48673a141b8b4991d96ec417c94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:39.659094 8193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ebb0b5d13874128a7628dbea53f7035. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:39.667017 8277 tablet_bootstrap.cc:492] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b: Bootstrap starting.
I20260923 17:39:39.669242 8278 tablet_bootstrap.cc:492] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d: Bootstrap starting.
I20260923 17:39:39.670984 8277 tablet_bootstrap.cc:654] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:39.672324 8277 log.cc:824] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:39.672840 8278 tablet_bootstrap.cc:654] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:39.673394 8279 tablet_bootstrap.cc:492] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497: Bootstrap starting.
I20260923 17:39:39.674225 8278 log.cc:824] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:39.676591 8279 tablet_bootstrap.cc:654] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:39.677573 8279 log.cc:824] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:39.680509 8279 tablet_bootstrap.cc:492] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497: No bootstrap required, opened a new log
I20260923 17:39:39.680496 8278 tablet_bootstrap.cc:492] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d: No bootstrap required, opened a new log
I20260923 17:39:39.680774 8279 ts_tablet_manager.cc:1397] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260923 17:39:39.680789 8278 ts_tablet_manager.cc:1397] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260923 17:39:39.682565 8277 tablet_bootstrap.cc:492] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b: No bootstrap required, opened a new log
I20260923 17:39:39.682860 8277 ts_tablet_manager.cc:1397] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260923 17:39:39.689145 8279 raft_consensus.cc:348] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9349631a01314397b09e1b7dafb2062b" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34707 } } peers { permanent_uuid: "3177e6b3fcd643cb96d0a9b733f6719d" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42797 } } peers { permanent_uuid: "422c0e771d3e45b39f670f77506dc497" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36837 } }
I20260923 17:39:39.689770 8279 raft_consensus.cc:374] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:39.689885 8279 raft_consensus.cc:729] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422c0e771d3e45b39f670f77506dc497, State: Initialized, Role: FOLLOWER
I20260923 17:39:39.690618 8279 consensus_queue.cc:260] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9349631a01314397b09e1b7dafb2062b" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34707 } } peers { permanent_uuid: "3177e6b3fcd643cb96d0a9b733f6719d" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42797 } } peers { permanent_uuid: "422c0e771d3e45b39f670f77506dc497" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36837 } }
I20260923 17:39:39.691794 8259 heartbeater.cc:503] Master 127.20.240.126:41501 was elected leader, sending a full tablet report...
I20260923 17:39:39.691712 8278 raft_consensus.cc:348] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9349631a01314397b09e1b7dafb2062b" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34707 } } peers { permanent_uuid: "3177e6b3fcd643cb96d0a9b733f6719d" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42797 } } peers { permanent_uuid: "422c0e771d3e45b39f670f77506dc497" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36837 } }
I20260923 17:39:39.692268 8278 raft_consensus.cc:374] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:39.692368 8278 raft_consensus.cc:729] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3177e6b3fcd643cb96d0a9b733f6719d, State: Initialized, Role: FOLLOWER
I20260923 17:39:39.693018 8278 consensus_queue.cc:260] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9349631a01314397b09e1b7dafb2062b" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34707 } } peers { permanent_uuid: "3177e6b3fcd643cb96d0a9b733f6719d" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42797 } } peers { permanent_uuid: "422c0e771d3e45b39f670f77506dc497" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36837 } }
I20260923 17:39:39.693933 8279 ts_tablet_manager.cc:1428] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260923 17:39:39.693869 8277 raft_consensus.cc:348] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9349631a01314397b09e1b7dafb2062b" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34707 } } peers { permanent_uuid: "3177e6b3fcd643cb96d0a9b733f6719d" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42797 } } peers { permanent_uuid: "422c0e771d3e45b39f670f77506dc497" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36837 } }
I20260923 17:39:39.694340 8277 raft_consensus.cc:374] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:39.694450 8277 raft_consensus.cc:729] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9349631a01314397b09e1b7dafb2062b, State: Initialized, Role: FOLLOWER
I20260923 17:39:39.695052 8128 heartbeater.cc:503] Master 127.20.240.126:41501 was elected leader, sending a full tablet report...
I20260923 17:39:39.695703 8278 ts_tablet_manager.cc:1428] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260923 17:39:39.696270 8277 consensus_queue.cc:260] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9349631a01314397b09e1b7dafb2062b" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34707 } } peers { permanent_uuid: "3177e6b3fcd643cb96d0a9b733f6719d" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42797 } } peers { permanent_uuid: "422c0e771d3e45b39f670f77506dc497" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36837 } }
I20260923 17:39:39.698333 8277 ts_tablet_manager.cc:1428] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260923 17:39:39.830726 8285 raft_consensus.cc:482] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:39:39.831193 8285 raft_consensus.cc:504] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9349631a01314397b09e1b7dafb2062b" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34707 } } peers { permanent_uuid: "3177e6b3fcd643cb96d0a9b733f6719d" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42797 } } peers { permanent_uuid: "422c0e771d3e45b39f670f77506dc497" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36837 } }
I20260923 17:39:39.833019 8285 leader_election.cc:288] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3177e6b3fcd643cb96d0a9b733f6719d (127.20.240.66:42797), 422c0e771d3e45b39f670f77506dc497 (127.20.240.67:36837)
W20260923 17:39:39.833299 8260 tablet.cc:2369] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:39:39.840302 8286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.833400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:36837 (local address 127.20.240.65:47917)
0923 17:39:39.834407 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:39.834438 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:39.834629 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:39.834998 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:39.835003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:39.835027 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:39.835531 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:39.835546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.836610 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.836616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:39.838242 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:39.838256 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.839510 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.839542 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:39.839715 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.839749 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:39:39.839834 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 17:39:39.839950 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":571,"threads_started":1}
I20260923 17:39:39.840404 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.833683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:54463 (local address 127.20.240.66:42797)
0923 17:39:39.834408 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:39.834415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:39.834861 (+ 446us) server_negotiation.cc:407] Connection header received
0923 17:39:39.835120 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:39.835126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:39.835244 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:39.835337 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:39.835969 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.836730 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.838636 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.839500 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.839670 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.839737 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:39.840000 (+ 263us) server_negotiation.cc:299] Negotiation successful
0923 17:39:39.840136 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554}
I20260923 17:39:39.840453 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.833659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:47917 (local address 127.20.240.67:36837)
0923 17:39:39.833866 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:39.833872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:39.834562 (+ 690us) server_negotiation.cc:407] Connection header received
0923 17:39:39.834783 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:39.834788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:39.834869 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:39.834956 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:39.835675 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.836465 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.838454 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.839316 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.839457 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.839511 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:39.839902 (+ 391us) server_negotiation.cc:299] Negotiation successful
0923 17:39:39.840040 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:39:39.840413 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.833412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:42797 (local address 127.20.240.65:54463)
0923 17:39:39.834746 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:39.834762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:39.834972 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:39.835351 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:39.835379 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:39.835406 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:39.835762 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:39.835775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.836963 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.836966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:39.838428 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:39.838442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.839692 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.839697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:39.839822 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.839862 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 17:39:39.839951 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 17:39:39.840072 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1220,"mutex_wait_us":476,"thread_start_us":514,"threads_started":1}
I20260923 17:39:39.841530 8082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebb0b5d13874128a7628dbea53f7035" candidate_uuid: "9349631a01314397b09e1b7dafb2062b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3177e6b3fcd643cb96d0a9b733f6719d" is_pre_election: true
I20260923 17:39:39.841522 8213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebb0b5d13874128a7628dbea53f7035" candidate_uuid: "9349631a01314397b09e1b7dafb2062b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422c0e771d3e45b39f670f77506dc497" is_pre_election: true
I20260923 17:39:39.842010 8082 raft_consensus.cc:2454] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9349631a01314397b09e1b7dafb2062b in term 0.
I20260923 17:39:39.842010 8213 raft_consensus.cc:2454] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9349631a01314397b09e1b7dafb2062b in term 0.
I20260923 17:39:39.842657 7886 leader_election.cc:302] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422c0e771d3e45b39f670f77506dc497, 9349631a01314397b09e1b7dafb2062b; no voters:
I20260923 17:39:39.843040 8285 raft_consensus.cc:2781] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:39:39.843225 8285 raft_consensus.cc:482] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:39:39.843333 8285 raft_consensus.cc:3037] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:39.845485 8285 raft_consensus.cc:504] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9349631a01314397b09e1b7dafb2062b" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34707 } } peers { permanent_uuid: "3177e6b3fcd643cb96d0a9b733f6719d" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42797 } } peers { permanent_uuid: "422c0e771d3e45b39f670f77506dc497" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36837 } }
I20260923 17:39:39.846148 8285 leader_election.cc:288] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [CANDIDATE]: Term 1 election: Requested vote from peers 3177e6b3fcd643cb96d0a9b733f6719d (127.20.240.66:42797), 422c0e771d3e45b39f670f77506dc497 (127.20.240.67:36837)
I20260923 17:39:39.846726 8213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebb0b5d13874128a7628dbea53f7035" candidate_uuid: "9349631a01314397b09e1b7dafb2062b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422c0e771d3e45b39f670f77506dc497"
I20260923 17:39:39.846728 8082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ebb0b5d13874128a7628dbea53f7035" candidate_uuid: "9349631a01314397b09e1b7dafb2062b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3177e6b3fcd643cb96d0a9b733f6719d"
I20260923 17:39:39.846959 8213 raft_consensus.cc:3037] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:39.847066 8082 raft_consensus.cc:3037] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:39.850281 8082 raft_consensus.cc:2454] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9349631a01314397b09e1b7dafb2062b in term 1.
I20260923 17:39:39.850281 8213 raft_consensus.cc:2454] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9349631a01314397b09e1b7dafb2062b in term 1.
I20260923 17:39:39.850780 7886 leader_election.cc:302] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422c0e771d3e45b39f670f77506dc497, 9349631a01314397b09e1b7dafb2062b; no voters:
I20260923 17:39:39.851221 8285 raft_consensus.cc:2781] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:39.851684 8285 raft_consensus.cc:686] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [term 1 LEADER]: Becoming Leader. State: Replica: 9349631a01314397b09e1b7dafb2062b, State: Running, Role: LEADER
I20260923 17:39:39.852160 8285 consensus_queue.cc:237] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9349631a01314397b09e1b7dafb2062b" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34707 } } peers { permanent_uuid: "3177e6b3fcd643cb96d0a9b733f6719d" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42797 } } peers { permanent_uuid: "422c0e771d3e45b39f670f77506dc497" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36837 } }
I20260923 17:39:39.856851 7808 catalog_manager.cc:5495] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9349631a01314397b09e1b7dafb2062b (127.20.240.65). New cstate: current_term: 1 leader_uuid: "9349631a01314397b09e1b7dafb2062b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9349631a01314397b09e1b7dafb2062b" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 34707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3177e6b3fcd643cb96d0a9b733f6719d" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "422c0e771d3e45b39f670f77506dc497" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36837 } health_report { overall_health: UNKNOWN } } }
W20260923 17:39:39.859397 8129 tablet.cc:2369] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:39:39.878057 7997 tablet.cc:2369] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:39:39.887483 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.882364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.20.240.65:34707)
0923 17:39:39.882565 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:39.882570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:39.882588 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:39:39.882678 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:39.882681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:39.882773 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:39.882926 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:39:39.883333 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.884064 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.885447 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.885856 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.885949 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.886452 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:39:39.886925 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:39:39.887083 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:39.887190 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 17:39:39.887228 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:39:39.895750 8213 raft_consensus.cc:1261] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497 [term 1 FOLLOWER]: Refusing update from remote peer 9349631a01314397b09e1b7dafb2062b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:39:39.895752 8082 raft_consensus.cc:1261] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d [term 1 FOLLOWER]: Refusing update from remote peer 9349631a01314397b09e1b7dafb2062b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:39:39.896684 8288 consensus_queue.cc:1035] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [LEADER]: Connected to new peer: Peer: permanent_uuid: "422c0e771d3e45b39f670f77506dc497" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:39:39.897039 8285 consensus_queue.cc:1035] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3177e6b3fcd643cb96d0a9b733f6719d" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 42797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:39:39.908589 8291 mvcc.cc:205] Tried to move back new op lower bound from 7332598496843059200 to 7332598496678105088. Current Snapshot: MvccSnapshot[applied={T|T < 7332598496843059200}]
I20260923 17:39:39.910358 8292 mvcc.cc:205] Tried to move back new op lower bound from 7332598496843059200 to 7332598496678105088. Current Snapshot: MvccSnapshot[applied={T|T < 7332598496843059200}]
I20260923 17:39:39.912865 8293 mvcc.cc:205] Tried to move back new op lower bound from 7332598496843059200 to 7332598496678105088. Current Snapshot: MvccSnapshot[applied={T|T < 7332598496843059200}]
I20260923 17:39:39.927855 7808 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:39:39.928160 7808 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:39:39.932575 7808 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c57762f419dd42dca175d93cac8d2be7: Sending DeleteTablet for 3 replicas of tablet 9ebb0b5d13874128a7628dbea53f7035
I20260923 17:39:39.934190 8062 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ebb0b5d13874128a7628dbea53f7035 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:39 UTC) from {username='slave'} at 127.0.0.1:58000
I20260923 17:39:39.934190 7930 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ebb0b5d13874128a7628dbea53f7035 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:39 UTC) from {username='slave'} at 127.0.0.1:43238
I20260923 17:39:39.934257 8193 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ebb0b5d13874128a7628dbea53f7035 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:39 UTC) from {username='slave'} at 127.0.0.1:47956
I20260923 17:39:39.936200 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 7868
I20260923 17:39:39.944413 8301 tablet_replica.cc:330] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b: stopping tablet replica
I20260923 17:39:39.945190 8301 raft_consensus.cc:2229] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:39:39.946168 8301 raft_consensus.cc:2258] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:39.947628 8303 tablet_replica.cc:330] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497: stopping tablet replica
I20260923 17:39:39.948158 8303 raft_consensus.cc:2229] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:39:39.948733 8303 raft_consensus.cc:2258] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:39.951354 8303 ts_tablet_manager.cc:1905] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:39.951953 8302 tablet_replica.cc:330] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d: stopping tablet replica
I20260923 17:39:39.952421 8302 raft_consensus.cc:2229] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:39:39.952998 8302 raft_consensus.cc:2258] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:39.955349 8302 ts_tablet_manager.cc:1905] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:39.959869 8303 ts_tablet_manager.cc:1918] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:39.960071 8303 log.cc:1196] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/9ebb0b5d13874128a7628dbea53f7035
I20260923 17:39:39.960599 8303 ts_tablet_manager.cc:1939] T 9ebb0b5d13874128a7628dbea53f7035 P 422c0e771d3e45b39f670f77506dc497: Deleting consensus metadata
I20260923 17:39:39.962689 7795 catalog_manager.cc:4841] TS 422c0e771d3e45b39f670f77506dc497 (127.20.240.67:36837): tablet 9ebb0b5d13874128a7628dbea53f7035 (table test-table [id=fa13f48673a141b8b4991d96ec417c94]) successfully deleted
I20260923 17:39:39.953055 8301 ts_tablet_manager.cc:1905] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:39.963332 8302 ts_tablet_manager.cc:1918] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:39.963498 8302 log.cc:1196] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/9ebb0b5d13874128a7628dbea53f7035
I20260923 17:39:39.964064 8302 ts_tablet_manager.cc:1939] T 9ebb0b5d13874128a7628dbea53f7035 P 3177e6b3fcd643cb96d0a9b733f6719d: Deleting consensus metadata
I20260923 17:39:39.965783 7792 catalog_manager.cc:4841] TS 3177e6b3fcd643cb96d0a9b733f6719d (127.20.240.66:42797): tablet 9ebb0b5d13874128a7628dbea53f7035 (table test-table [id=fa13f48673a141b8b4991d96ec417c94]) successfully deleted
I20260923 17:39:39.970181 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:39.937358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.20.240.65:34707)
0923 17:39:39.937560 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:39.937566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:39.953383 (+ 15817us) server_negotiation.cc:407] Connection header received
0923 17:39:39.953923 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:39.953929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:39.954043 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:39.954143 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:39.954927 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.960732 (+ 5805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.963136 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:39.963878 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:39.964027 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:39.968518 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:39.968532 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:39.968546 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:39.969282 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:39.969344 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:39.969374 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:39.969490 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:39.969675 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 17:39:39.969788 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:39:39.970854 7991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:40.071327 8301 ts_tablet_manager.cc:1918] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:40.071630 8301 log.cc:1196] T 9ebb0b5d13874128a7628dbea53f7035 P 9349631a01314397b09e1b7dafb2062b: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/9ebb0b5d13874128a7628dbea53f7035
I20260923 17:39:40.072027 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 7868
I20260923 17:39:40.088795 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 8000
W20260923 17:39:40.088914 7795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.240.65:34707 (error 108)
I20260923 17:39:40.090855 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:40.090352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34707 (local address 127.0.0.1:43280)
0923 17:39:40.090545 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:40.090677 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 17:39:40.091107 7795 catalog_manager.cc:4584] TS 9349631a01314397b09e1b7dafb2062b (127.20.240.65:34707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ebb0b5d13874128a7628dbea53f7035: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34707: connect: Connection refused (error 111)
I20260923 17:39:40.096081 8274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:40.089655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.20.240.66:42797)
0923 17:39:40.089836 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:40.089841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:40.089890 (+ 49us) server_negotiation.cc:407] Connection header received
0923 17:39:40.090099 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:40.090103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:40.090215 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:40.090359 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:40.090826 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:40.091783 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:40.093173 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:40.093604 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:40.093698 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:40.094715 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:40.094724 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:40.094738 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:40.095414 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:40.095465 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:40.095486 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:40.095581 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:40.095715 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 17:39:40.095803 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:39:40.096760 8123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:40.144969 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:40.144473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34707 (local address 127.0.0.1:43282)
0923 17:39:40.144701 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:40.144796 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 17:39:40.145955 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:40.145601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34707 (local address 127.0.0.1:43294)
0923 17:39:40.145749 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:40.145826 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 17:39:40.187486 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 8000
I20260923 17:39:40.202330 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 8131
I20260923 17:39:40.206386 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:40.205865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34707 (local address 127.0.0.1:43298)
0923 17:39:40.206097 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:40.206185 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260923 17:39:40.207469 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:40.207078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34707 (local address 127.0.0.1:43304)
0923 17:39:40.207264 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:40.207341 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 17:39:40.209296 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:40.203072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47964 (local address 127.20.240.67:36837)
0923 17:39:40.203300 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:40.203305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:40.203324 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:39:40.203487 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:40.203491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:40.203577 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:40.203667 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:40.204048 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:40.204768 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:40.206320 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:40.206868 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:40.206994 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:40.208083 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:40.208092 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:40.208108 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:40.208684 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:40.208733 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:40.208753 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:40.208843 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:40.208921 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 17:39:40.209008 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260923 17:39:40.209900 8254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:40.288228 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:40.287623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34707 (local address 127.0.0.1:43320)
0923 17:39:40.287879 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:40.288009 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260923 17:39:40.289552 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:40.289087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:34707 (local address 127.0.0.1:43322)
0923 17:39:40.289299 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:40.289394 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.65:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260923 17:39:40.298802 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 8131
I20260923 17:39:40.312347 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 7777
I20260923 17:39:40.313683 7839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:40.423488 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 7777
2026-09-23T17:39:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2886 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:39:40 dist-test-slave-ppjl krb5kdc[8312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:39:40 dist-test-slave-ppjl krb5kdc[8312](info): set up 2 sockets
Sep 23 17:39:40 dist-test-slave-ppjl krb5kdc[8312](info): commencing operation
krb5kdc: starting...
W20260923 17:39:41.452124 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:39:41 dist-test-slave-ppjl krb5kdc[8312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:39:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:39:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:39:41.702858 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41289
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:41289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:42.037900 8325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:42.038314 8325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:42.038470 8325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:42.046139 8325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:39:42.046294 8325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:42.046373 8325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:42.046456 8325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:39:42.046515 8325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:39:42.057184 8325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:41289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:42.058699 8325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:42.060112 8325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:42.072592 8333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:42.072693 8330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:42.072664 8331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:42.073706 8325 server_base.cc:1034] running on GCE node
I20260923 17:39:42.074687 8325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:42.076421 8325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:42.077656 8325 hybrid_clock.cc:648] HybridClock initialized: now 1790185182077612 us; error 31 us; skew 500 ppm
I20260923 17:39:42.078115 8325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:42 dist-test-slave-ppjl krb5kdc[8312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185182, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:42.088290 8325 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:39:42.090035 8325 webserver.cc:460] Webserver started at http://127.20.240.126:40895/ using document root <none> and password file <none>
I20260923 17:39:42.090827 8325 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:42.090955 8325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:42.091364 8325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:42.094779 8325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "444f233eb0964e95aeb25ca99979b663"
format_stamp: "Formatted at 2026-09-23 17:39:42 on dist-test-slave-ppjl"
server_key: "c003874f014862a7acb9e26aedf1df8c"
server_key_iv: "7fac74efe50c9d0572d135d861572f14"
server_key_version: "encryptionkey@0"
I20260923 17:39:42.095706 8325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "444f233eb0964e95aeb25ca99979b663"
format_stamp: "Formatted at 2026-09-23 17:39:42 on dist-test-slave-ppjl"
server_key: "c003874f014862a7acb9e26aedf1df8c"
server_key_iv: "7fac74efe50c9d0572d135d861572f14"
server_key_version: "encryptionkey@0"
I20260923 17:39:42.101629 8325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:39:42.105639 8340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:42.107471 8325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:39:42.107656 8325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "444f233eb0964e95aeb25ca99979b663"
format_stamp: "Formatted at 2026-09-23 17:39:42 on dist-test-slave-ppjl"
server_key: "c003874f014862a7acb9e26aedf1df8c"
server_key_iv: "7fac74efe50c9d0572d135d861572f14"
server_key_version: "encryptionkey@0"
I20260923 17:39:42.107856 8325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:42.128137 8325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:42.132814 8325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:42.133117 8325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:42.159787 8325 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:41289
I20260923 17:39:42.159871 8391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:41289 every 8 connection(s)
I20260923 17:39:42.161212 8325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:39:42.165223 8392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:42.165799 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 8325
I20260923 17:39:42.166003 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:39:42.166527 21441 external_mini_cluster.cc:1428] Setting key ea29ad652b62488d8693c840c7dbf5a6
I20260923 17:39:42.180886 8392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663: Bootstrap starting.
Sep 23 17:39:42 dist-test-slave-ppjl krb5kdc[8312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:39:42.185189 8392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:42.186327 8392 log.cc:824] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:42.189769 8392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663: No bootstrap required, opened a new log
I20260923 17:39:42.192791 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:42.168572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.20.240.126:41289)
0923 17:39:42.169377 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:42.169389 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:42.169438 (+ 49us) server_negotiation.cc:407] Connection header received
0923 17:39:42.171712 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:42.171741 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:42.172300 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:42.172679 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:42.174874 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:42.176599 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:42.178810 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:42.179664 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:42.179824 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:42.184946 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:42.184955 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:42.184972 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:42.185018 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:42.188233 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:42.189228 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:42.189235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:42.189241 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:42.189453 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:42.189940 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:42.189945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:42.189948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:42.190536 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:42.190769 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:42.191282 (+ 513us) server_negotiation.cc:299] Negotiation successful
0923 17:39:42.191703 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":590,"thread_start_us":374,"threads_started":1}
I20260923 17:39:42.198464 8392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444f233eb0964e95aeb25ca99979b663" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41289 } }
I20260923 17:39:42.198904 8392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:42.198998 8392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444f233eb0964e95aeb25ca99979b663, State: Initialized, Role: FOLLOWER
I20260923 17:39:42.199651 8392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444f233eb0964e95aeb25ca99979b663" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41289 } }
I20260923 17:39:42.199891 8392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:39:42.200026 8392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:39:42.200176 8392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:42.202631 8392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444f233eb0964e95aeb25ca99979b663" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41289 } }
I20260923 17:39:42.203137 8392 leader_election.cc:302] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 444f233eb0964e95aeb25ca99979b663; no voters:
I20260923 17:39:42.203789 8392 leader_election.cc:288] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:39:42.203944 8397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:42.204622 8397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [term 1 LEADER]: Becoming Leader. State: Replica: 444f233eb0964e95aeb25ca99979b663, State: Running, Role: LEADER
I20260923 17:39:42.205191 8397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444f233eb0964e95aeb25ca99979b663" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41289 } }
I20260923 17:39:42.205847 8392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:39:42.208022 8398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "444f233eb0964e95aeb25ca99979b663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444f233eb0964e95aeb25ca99979b663" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41289 } } }
I20260923 17:39:42.208446 8398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:42.209304 8405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:39:42.209162 8399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 444f233eb0964e95aeb25ca99979b663. Latest consensus state: current_term: 1 leader_uuid: "444f233eb0964e95aeb25ca99979b663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "444f233eb0964e95aeb25ca99979b663" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41289 } } }
I20260923 17:39:42.209548 8399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663 [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:42.214570 8405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:39:42.224080 8405 catalog_manager.cc:1347] Generated new cluster ID: 6a504532d5584b7d937fe96aa581352a
I20260923 17:39:42.224205 8405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:39:42.264933 8405 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:39:42.267064 8405 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:39:42.275947 8405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663: Generated new TSK 0
I20260923 17:39:42.277086 8405 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
Entry for principal kudu/127.20.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab.
I20260923 17:39:42.340981 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41289
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:42.678968 8418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:42.679346 8418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:42.679502 8418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:42.687019 8418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:42.687184 8418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:42.687366 8418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:39:42.698832 8418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.65.keytab
--principal=kudu/127.20.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:42.700265 8418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:42.701665 8418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:42.715059 8423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:42.715087 8424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:42.715137 8426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:42.716292 8418 server_base.cc:1034] running on GCE node
I20260923 17:39:42.717011 8418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:42.718014 8418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:42.719249 8418 hybrid_clock.cc:648] HybridClock initialized: now 1790185182719198 us; error 34 us; skew 500 ppm
I20260923 17:39:42.719583 8418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:42 dist-test-slave-ppjl krb5kdc[8312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185182, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:42.723110 8418 init.cc:377] Logged in from keytab as kudu/127.20.240.65@KRBTEST.COM (short username kudu)
I20260923 17:39:42.724718 8418 webserver.cc:460] Webserver started at http://127.20.240.65:38625/ using document root <none> and password file <none>
I20260923 17:39:42.725412 8418 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:42.725518 8418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:42.725797 8418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:42.728914 8418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "3e732c84b2254e19a3653f66359f0c55"
format_stamp: "Formatted at 2026-09-23 17:39:42 on dist-test-slave-ppjl"
server_key: "423971deee2acbd693715971609ba1dd"
server_key_iv: "2ec9411c8f8b0b00ace6daeac42dc384"
server_key_version: "encryptionkey@0"
I20260923 17:39:42.729697 8418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "3e732c84b2254e19a3653f66359f0c55"
format_stamp: "Formatted at 2026-09-23 17:39:42 on dist-test-slave-ppjl"
server_key: "423971deee2acbd693715971609ba1dd"
server_key_iv: "2ec9411c8f8b0b00ace6daeac42dc384"
server_key_version: "encryptionkey@0"
I20260923 17:39:42.735219 8418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 17:39:42.739377 8433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:42.740996 8418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:39:42.741158 8418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "3e732c84b2254e19a3653f66359f0c55"
format_stamp: "Formatted at 2026-09-23 17:39:42 on dist-test-slave-ppjl"
server_key: "423971deee2acbd693715971609ba1dd"
server_key_iv: "2ec9411c8f8b0b00ace6daeac42dc384"
server_key_version: "encryptionkey@0"
I20260923 17:39:42.741362 8418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:42.764566 8418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:42.769050 8418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:42.769306 8418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:42.770321 8418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:42.771715 8418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:42.771891 8418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:42.772001 8418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:42.772045 8418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:42.802923 8418 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:37755
I20260923 17:39:42.802963 8545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:37755 every 8 connection(s)
I20260923 17:39:42.804318 8418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:39:42.809484 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 8418
I20260923 17:39:42.809633 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:39:42.810107 21441 external_mini_cluster.cc:1428] Setting key 68135bf4c400e1fcb95b735b4ab18bf7
Sep 23 17:39:42 dist-test-slave-ppjl krb5kdc[8312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185182, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.65@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:39:42.826001 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:42.806992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:42179 (local address 127.20.240.126:41289)
0923 17:39:42.807503 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:42.807507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:42.808481 (+ 974us) server_negotiation.cc:407] Connection header received
0923 17:39:42.810528 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:42.810535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:42.810639 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:42.810758 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:42.812504 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:42.813525 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:42.815138 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:42.815698 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:42.815826 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:42.819600 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:42.819606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:42.819611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:42.819653 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:42.822575 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:42.823572 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:42.823577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:42.823579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:42.823714 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:42.824199 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:42.824202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:42.824204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:42.824487 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:42.824600 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:42.825507 (+ 907us) server_negotiation.cc:299] Negotiation successful
0923 17:39:42.825713 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":277,"threads_started":1}
I20260923 17:39:42.827709 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:42.807494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41289 (local address 127.20.240.65:42179)
0923 17:39:42.808296 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:42.808338 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:42.810236 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:42.811034 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:42.811044 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:42.811663 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:42.812304 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:42.812343 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:42.813701 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:42.813705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:42.814991 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:42.815005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:42.815916 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:42.815921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:42.816130 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:42.816867 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:39:42.816896 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:42.819354 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:42.822746 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:42.822757 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:42.822776 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:42.823435 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:42.823833 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:42.823838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:42.823851 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:42.824093 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:42.824599 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:42.824607 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:42.825082 (+ 475us) client_negotiation.cc:769] Sending connection context
0923 17:39:42.825374 (+ 292us) client_negotiation.cc:240] Negotiation successful
0923 17:39:42.825969 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":343,"threads_started":1}
I20260923 17:39:42.829556 8546 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41289
I20260923 17:39:42.829890 8546 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:42.830665 8546 heartbeater.cc:511] Master 127.20.240.126:41289 requested a full tablet report, sending...
I20260923 17:39:42.833117 8357 ts_manager.cc:194] Registered new tserver with Master: 3e732c84b2254e19a3653f66359f0c55 (127.20.240.65:37755)
I20260923 17:39:42.834625 8357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.65@KRBTEST.COM'} at 127.20.240.65:42179
WARNING: no policy specified for kudu/127.20.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
Entry for principal kudu/127.20.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab.
I20260923 17:39:42.866781 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41289
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:43.195240 8552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:43.195578 8552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:43.195713 8552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:43.203271 8552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:43.203403 8552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:43.203562 8552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:39:43.213917 8552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.66.keytab
--principal=kudu/127.20.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:43.215293 8552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:43.216655 8552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:43.231768 8557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:43.231812 8558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:43.231736 8561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:43.232722 8552 server_base.cc:1034] running on GCE node
I20260923 17:39:43.233413 8552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:43.234462 8552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:43.235702 8552 hybrid_clock.cc:648] HybridClock initialized: now 1790185183235642 us; error 58 us; skew 500 ppm
I20260923 17:39:43.236083 8552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:43 dist-test-slave-ppjl krb5kdc[8312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185183, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:43.239543 8552 init.cc:377] Logged in from keytab as kudu/127.20.240.66@KRBTEST.COM (short username kudu)
I20260923 17:39:43.241369 8552 webserver.cc:460] Webserver started at http://127.20.240.66:42861/ using document root <none> and password file <none>
I20260923 17:39:43.242105 8552 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:43.242230 8552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:43.242544 8552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:43.245651 8552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "da15875b91014f2f8ae59a6e558bef87"
format_stamp: "Formatted at 2026-09-23 17:39:43 on dist-test-slave-ppjl"
server_key: "1f1b98202b627b23b0fa40b664ddbc6c"
server_key_iv: "3a49bdadfbac276461586e95cd93702d"
server_key_version: "encryptionkey@0"
I20260923 17:39:43.246421 8552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "da15875b91014f2f8ae59a6e558bef87"
format_stamp: "Formatted at 2026-09-23 17:39:43 on dist-test-slave-ppjl"
server_key: "1f1b98202b627b23b0fa40b664ddbc6c"
server_key_iv: "3a49bdadfbac276461586e95cd93702d"
server_key_version: "encryptionkey@0"
I20260923 17:39:43.251969 8552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:39:43.255576 8568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:43.257068 8552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:39:43.257231 8552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "da15875b91014f2f8ae59a6e558bef87"
format_stamp: "Formatted at 2026-09-23 17:39:43 on dist-test-slave-ppjl"
server_key: "1f1b98202b627b23b0fa40b664ddbc6c"
server_key_iv: "3a49bdadfbac276461586e95cd93702d"
server_key_version: "encryptionkey@0"
I20260923 17:39:43.257447 8552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:43.281040 8552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:43.285892 8552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:43.286185 8552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:43.287277 8552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:43.288731 8552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:43.288820 8552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:43.288955 8552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:43.289008 8552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:43.320549 8552 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:45809
I20260923 17:39:43.320576 8682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:45809 every 8 connection(s)
I20260923 17:39:43.321986 8552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:39:43.331120 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 8552
I20260923 17:39:43.331367 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:39:43.331845 21441 external_mini_cluster.cc:1428] Setting key 3531b20a014851099ad06a9c4ef79646
Sep 23 17:39:43 dist-test-slave-ppjl krb5kdc[8312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185183, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.66@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:39:43.344538 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:43.324797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:53995 (local address 127.20.240.126:41289)
0923 17:39:43.325000 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:43.325005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:43.326104 (+ 1099us) server_negotiation.cc:407] Connection header received
0923 17:39:43.328187 (+ 2083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:43.328194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:43.328294 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:43.328398 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:43.330119 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.330948 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.332775 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.333350 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.333463 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:43.337144 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:43.337151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:43.337156 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:43.337203 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:43.340697 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:43.341585 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:43.341591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:43.341594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:43.341793 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:43.342329 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:43.342334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:43.342338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:43.342739 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:43.342931 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:43.344053 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0923 17:39:43.344261 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:39:43.345880 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:43.325144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41289 (local address 127.20.240.66:53995)
0923 17:39:43.325920 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:43.325964 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:43.327864 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:43.328673 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:43.328682 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:43.329277 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:43.329907 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:43.329928 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.331134 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.331139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:43.332620 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:43.332633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.333619 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.333626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:43.333877 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:43.334552 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:39:43.334578 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:43.336844 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:43.340914 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:43.340923 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:43.340940 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:43.341422 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:43.341961 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:43.341965 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:43.341968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:43.342191 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:43.342952 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:43.342962 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:43.343519 (+ 557us) client_negotiation.cc:769] Sending connection context
0923 17:39:43.343851 (+ 332us) client_negotiation.cc:240] Negotiation successful
0923 17:39:43.344442 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":291,"threads_started":1}
I20260923 17:39:43.347642 8683 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41289
I20260923 17:39:43.348062 8683 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:43.348840 8683 heartbeater.cc:511] Master 127.20.240.126:41289 requested a full tablet report, sending...
I20260923 17:39:43.350368 8357 ts_manager.cc:194] Registered new tserver with Master: da15875b91014f2f8ae59a6e558bef87 (127.20.240.66:45809)
I20260923 17:39:43.351233 8357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.66@KRBTEST.COM'} at 127.20.240.66:53995
WARNING: no policy specified for kudu/127.20.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
Entry for principal kudu/127.20.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab.
I20260923 17:39:43.388669 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41289
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:43.713980 8688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:43.714298 8688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:43.714447 8688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:43.721824 8688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:43.721971 8688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:43.722131 8688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:39:43.732277 8688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.67.keytab
--principal=kudu/127.20.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:41289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:43.733633 8688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:43.734999 8688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:43.748677 8694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:43.748759 8693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:43.748924 8696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:43.749562 8688 server_base.cc:1034] running on GCE node
I20260923 17:39:43.750196 8688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:43.751188 8688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:43.752427 8688 hybrid_clock.cc:648] HybridClock initialized: now 1790185183752373 us; error 41 us; skew 500 ppm
I20260923 17:39:43.752754 8688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:43 dist-test-slave-ppjl krb5kdc[8312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185183, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:43.756031 8688 init.cc:377] Logged in from keytab as kudu/127.20.240.67@KRBTEST.COM (short username kudu)
I20260923 17:39:43.757808 8688 webserver.cc:460] Webserver started at http://127.20.240.67:33669/ using document root <none> and password file <none>
I20260923 17:39:43.758528 8688 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:43.758656 8688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:43.758975 8688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:43.762172 8688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "350133d188d843ed8f21e96de5cd1ba6"
format_stamp: "Formatted at 2026-09-23 17:39:43 on dist-test-slave-ppjl"
server_key: "9fee86455d95a57af9f50f35835d27f7"
server_key_iv: "cb11a5678970806bcac492e0b3988522"
server_key_version: "encryptionkey@0"
I20260923 17:39:43.762961 8688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "350133d188d843ed8f21e96de5cd1ba6"
format_stamp: "Formatted at 2026-09-23 17:39:43 on dist-test-slave-ppjl"
server_key: "9fee86455d95a57af9f50f35835d27f7"
server_key_iv: "cb11a5678970806bcac492e0b3988522"
server_key_version: "encryptionkey@0"
I20260923 17:39:43.768676 8688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:39:43.772528 8703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:43.774081 8688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 17:39:43.774243 8688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "350133d188d843ed8f21e96de5cd1ba6"
format_stamp: "Formatted at 2026-09-23 17:39:43 on dist-test-slave-ppjl"
server_key: "9fee86455d95a57af9f50f35835d27f7"
server_key_iv: "cb11a5678970806bcac492e0b3988522"
server_key_version: "encryptionkey@0"
I20260923 17:39:43.774446 8688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:43.817648 8688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:43.822324 8688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:43.822638 8688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:43.823668 8688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:43.825053 8688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:43.825129 8688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:43.825222 8688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:43.825281 8688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:43.838076 8546 heartbeater.cc:503] Master 127.20.240.126:41289 was elected leader, sending a full tablet report...
I20260923 17:39:43.857280 8688 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:45191
I20260923 17:39:43.857318 8816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:45191 every 8 connection(s)
I20260923 17:39:43.858623 8688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:39:43.863560 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 8688
I20260923 17:39:43.863739 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:39:43.864187 21441 external_mini_cluster.cc:1428] Setting key b5c4ac6f77bf8f50d3df251fa9770ddd
Sep 23 17:39:43 dist-test-slave-ppjl krb5kdc[8312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185183, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.67@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:39:43.882035 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:43.861490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:41625 (local address 127.20.240.126:41289)
0923 17:39:43.862027 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:43.862032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:43.863122 (+ 1090us) server_negotiation.cc:407] Connection header received
0923 17:39:43.865305 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:43.865309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:43.865443 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:43.865554 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:43.867974 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.868843 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.870652 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.871148 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.871297 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:43.875064 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:43.875070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:43.875073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:43.875113 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:43.878212 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:43.879377 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:43.879381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:43.879384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:43.879524 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:43.880108 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:43.880112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:43.880114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:43.880434 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:43.880582 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:43.881409 (+ 827us) server_negotiation.cc:299] Negotiation successful
0923 17:39:43.881653 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":247,"threads_started":1}
I20260923 17:39:43.883095 8820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:43.862140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:41289 (local address 127.20.240.67:41625)
0923 17:39:43.862936 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:43.862980 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:43.864971 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:43.865993 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:43.866003 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:43.866694 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:43.867671 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:43.867701 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.869086 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.869091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:43.870472 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:43.870489 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.871443 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.871450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:43.871676 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:43.872450 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:39:43.872476 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:43.874767 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:43.878430 (+ 3663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:43.878440 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:43.878460 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:43.879206 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:43.879648 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:39:43.879653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:39:43.879658 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:39:43.879977 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:39:43.880577 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:43.880586 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:43.880982 (+ 396us) client_negotiation.cc:769] Sending connection context
0923 17:39:43.881250 (+ 268us) client_negotiation.cc:240] Negotiation successful
0923 17:39:43.881859 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":302,"threads_started":1}
I20260923 17:39:43.884835 8817 heartbeater.cc:347] Connected to a master server at 127.20.240.126:41289
I20260923 17:39:43.885178 8817 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:43.886034 8817 heartbeater.cc:511] Master 127.20.240.126:41289 requested a full tablet report, sending...
I20260923 17:39:43.887764 8357 ts_manager.cc:194] Registered new tserver with Master: 350133d188d843ed8f21e96de5cd1ba6 (127.20.240.67:45191)
I20260923 17:39:43.888842 8357 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.240.67@KRBTEST.COM'} at 127.20.240.67:41625
I20260923 17:39:43.893627 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:39:43.910714 8819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:43.897888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59130 (local address 127.20.240.126:41289)
0923 17:39:43.898095 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:43.898100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:43.898450 (+ 350us) server_negotiation.cc:407] Connection header received
0923 17:39:43.898605 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:43.898609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:43.898698 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:43.898813 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:43.900248 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.900986 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.902391 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.902822 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.902921 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:43.905333 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:43.905339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:43.905342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:43.905383 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:43.908069 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:43.908769 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:43.908773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:43.908782 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:43.908926 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:43.909263 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:43.909266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:43.909268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:43.909594 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:43.909777 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:43.910272 (+ 495us) server_negotiation.cc:299] Negotiation successful
0923 17:39:43.910466 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 17:39:43.915059 8357 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:39:43.922263 8357 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:39:43.949208 8831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:43.936102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.20.240.66:45809)
0923 17:39:43.940554 (+ 4452us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:43.940563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:43.940590 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:39:43.940679 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:43.940685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:43.940917 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:43.941685 (+ 768us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:43.941917 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.943393 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.945630 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.947504 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.947675 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:43.947929 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:43.948676 (+ 747us) server_negotiation.cc:299] Negotiation successful
0923 17:39:43.948861 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4293,"thread_start_us":4188,"threads_started":1}
I20260923 17:39:43.949229 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:43.937041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.20.240.67:45191)
0923 17:39:43.938515 (+ 1474us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:43.938525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:43.939757 (+ 1232us) server_negotiation.cc:407] Connection header received
0923 17:39:43.939995 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:43.940003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:43.940289 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:43.940507 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:43.941255 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.942800 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.946130 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.947568 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.947685 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:43.947954 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:43.948594 (+ 640us) server_negotiation.cc:299] Negotiation successful
0923 17:39:43.948885 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1287,"thread_start_us":1144,"threads_started":1}
I20260923 17:39:43.949391 8830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:43.935808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:45809 (local address 127.0.0.1:60758)
0923 17:39:43.936834 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:43.936876 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:43.937047 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:43.941133 (+ 4086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:43.941138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:43.941159 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:43.941545 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:43.941555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.943565 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.943570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:43.945410 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:43.945425 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.948112 (+ 2687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.948118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:43.948310 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:43.948367 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 17:39:43.948514 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 17:39:43.949032 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":344,"thread_start_us":1415,"threads_started":1}
I20260923 17:39:43.949959 8832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:43.936473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:45191 (local address 127.0.0.1:43532)
0923 17:39:43.939632 (+ 3159us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:43.939650 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:43.939763 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:43.940594 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:43.940601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:43.940631 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:43.941058 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:43.941073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.943006 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.943012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:43.945920 (+ 2908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:43.945929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.947778 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.947785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:43.948181 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:43.948369 (+ 188us) client_negotiation.cc:769] Sending connection context
0923 17:39:43.948515 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 17:39:43.948755 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3059,"mutex_wait_us":123,"thread_start_us":2911,"threads_started":1}
I20260923 17:39:43.950997 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:43.936444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:37755 (local address 127.0.0.1:47104)
0923 17:39:43.938996 (+ 2552us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:43.939017 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:43.939140 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:43.940911 (+ 1771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:43.940916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:43.940944 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:43.941429 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:43.941441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.943371 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.943376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:43.945907 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:43.945925 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.950164 (+ 4239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.950172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:43.950363 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:43.950397 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:39:43.950498 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 17:39:43.950640 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2412,"mutex_wait_us":75,"thread_start_us":2245,"threads_started":1}
I20260923 17:39:43.951064 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:43.937164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.20.240.65:37755)
0923 17:39:43.940316 (+ 3152us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:43.940324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:43.940350 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:39:43.940461 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:43.940468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:43.940703 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:43.940920 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:43.941643 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.943201 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.948076 (+ 4875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:43.949609 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:43.949752 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:43.950001 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:43.950582 (+ 581us) server_negotiation.cc:299] Negotiation successful
0923 17:39:43.950752 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2960,"thread_start_us":2794,"threads_started":1}
I20260923 17:39:43.954368 8752 tablet_service.cc:1467] Processing CreateTablet for tablet 3102d156d1b041eba3205f07c4c85433 (DEFAULT_TABLE table=test-table [id=4c456de580154f5fbfd0cefaadd00279]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:43.954365 8617 tablet_service.cc:1467] Processing CreateTablet for tablet 3102d156d1b041eba3205f07c4c85433 (DEFAULT_TABLE table=test-table [id=4c456de580154f5fbfd0cefaadd00279]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:43.956051 8752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3102d156d1b041eba3205f07c4c85433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:43.956051 8617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3102d156d1b041eba3205f07c4c85433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:43.959499 8481 tablet_service.cc:1467] Processing CreateTablet for tablet 3102d156d1b041eba3205f07c4c85433 (DEFAULT_TABLE table=test-table [id=4c456de580154f5fbfd0cefaadd00279]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:43.960937 8481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3102d156d1b041eba3205f07c4c85433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:43.973235 8838 tablet_bootstrap.cc:492] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55: Bootstrap starting.
I20260923 17:39:43.976646 8838 tablet_bootstrap.cc:654] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:43.977710 8838 log.cc:824] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:43.978209 8836 tablet_bootstrap.cc:492] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87: Bootstrap starting.
I20260923 17:39:43.978873 8837 tablet_bootstrap.cc:492] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6: Bootstrap starting.
I20260923 17:39:43.981573 8838 tablet_bootstrap.cc:492] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55: No bootstrap required, opened a new log
I20260923 17:39:43.981953 8838 ts_tablet_manager.cc:1397] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 17:39:43.982532 8836 tablet_bootstrap.cc:654] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:43.982785 8837 tablet_bootstrap.cc:654] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:43.983678 8836 log.cc:824] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:43.984064 8837 log.cc:824] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:43.986831 8837 tablet_bootstrap.cc:492] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6: No bootstrap required, opened a new log
I20260923 17:39:43.986955 8836 tablet_bootstrap.cc:492] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87: No bootstrap required, opened a new log
I20260923 17:39:43.987133 8837 ts_tablet_manager.cc:1397] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260923 17:39:43.987260 8836 ts_tablet_manager.cc:1397] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 17:39:43.992689 8838 raft_consensus.cc:348] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da15875b91014f2f8ae59a6e558bef87" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45809 } } peers { permanent_uuid: "3e732c84b2254e19a3653f66359f0c55" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37755 } } peers { permanent_uuid: "350133d188d843ed8f21e96de5cd1ba6" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45191 } }
I20260923 17:39:43.993243 8838 raft_consensus.cc:374] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:43.993396 8838 raft_consensus.cc:729] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e732c84b2254e19a3653f66359f0c55, State: Initialized, Role: FOLLOWER
I20260923 17:39:43.994103 8838 consensus_queue.cc:260] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da15875b91014f2f8ae59a6e558bef87" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45809 } } peers { permanent_uuid: "3e732c84b2254e19a3653f66359f0c55" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37755 } } peers { permanent_uuid: "350133d188d843ed8f21e96de5cd1ba6" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45191 } }
I20260923 17:39:43.995990 8837 raft_consensus.cc:348] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da15875b91014f2f8ae59a6e558bef87" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45809 } } peers { permanent_uuid: "3e732c84b2254e19a3653f66359f0c55" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37755 } } peers { permanent_uuid: "350133d188d843ed8f21e96de5cd1ba6" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45191 } }
I20260923 17:39:43.996527 8837 raft_consensus.cc:374] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:43.996639 8838 ts_tablet_manager.cc:1428] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260923 17:39:43.996693 8837 raft_consensus.cc:729] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 350133d188d843ed8f21e96de5cd1ba6, State: Initialized, Role: FOLLOWER
I20260923 17:39:43.997423 8837 consensus_queue.cc:260] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da15875b91014f2f8ae59a6e558bef87" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45809 } } peers { permanent_uuid: "3e732c84b2254e19a3653f66359f0c55" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37755 } } peers { permanent_uuid: "350133d188d843ed8f21e96de5cd1ba6" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45191 } }
I20260923 17:39:43.997665 8836 raft_consensus.cc:348] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da15875b91014f2f8ae59a6e558bef87" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45809 } } peers { permanent_uuid: "3e732c84b2254e19a3653f66359f0c55" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37755 } } peers { permanent_uuid: "350133d188d843ed8f21e96de5cd1ba6" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45191 } }
I20260923 17:39:43.998196 8836 raft_consensus.cc:374] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:43.998333 8836 raft_consensus.cc:729] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da15875b91014f2f8ae59a6e558bef87, State: Initialized, Role: FOLLOWER
I20260923 17:39:43.998653 8817 heartbeater.cc:503] Master 127.20.240.126:41289 was elected leader, sending a full tablet report...
I20260923 17:39:43.999425 8837 ts_tablet_manager.cc:1428] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260923 17:39:43.999292 8836 consensus_queue.cc:260] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da15875b91014f2f8ae59a6e558bef87" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45809 } } peers { permanent_uuid: "3e732c84b2254e19a3653f66359f0c55" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37755 } } peers { permanent_uuid: "350133d188d843ed8f21e96de5cd1ba6" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45191 } }
I20260923 17:39:44.000468 8683 heartbeater.cc:503] Master 127.20.240.126:41289 was elected leader, sending a full tablet report...
I20260923 17:39:44.001160 8836 ts_tablet_manager.cc:1428] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260923 17:39:44.045524 8844 raft_consensus.cc:482] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:39:44.045861 8844 raft_consensus.cc:504] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da15875b91014f2f8ae59a6e558bef87" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45809 } } peers { permanent_uuid: "3e732c84b2254e19a3653f66359f0c55" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37755 } } peers { permanent_uuid: "350133d188d843ed8f21e96de5cd1ba6" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45191 } }
I20260923 17:39:44.047698 8844 leader_election.cc:288] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e732c84b2254e19a3653f66359f0c55 (127.20.240.65:37755), 350133d188d843ed8f21e96de5cd1ba6 (127.20.240.67:45191)
I20260923 17:39:44.055337 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:44.048219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:50191 (local address 127.20.240.67:45191)
0923 17:39:44.048448 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:44.048456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:44.049316 (+ 860us) server_negotiation.cc:407] Connection header received
0923 17:39:44.049491 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:44.049495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:44.049555 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:44.049629 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:44.050251 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.051408 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.053427 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.054232 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.054421 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:44.054523 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:44.054928 (+ 405us) server_negotiation.cc:299] Negotiation successful
0923 17:39:44.055068 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 17:39:44.055354 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:44.048129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:47089 (local address 127.20.240.65:37755)
0923 17:39:44.048368 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:44.048376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:44.049180 (+ 804us) server_negotiation.cc:407] Connection header received
0923 17:39:44.049421 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:44.049425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:44.049488 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:44.049565 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:44.050182 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.051061 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.053338 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.054305 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.054476 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:44.054557 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:44.054910 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 17:39:44.055050 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:39:44.055558 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:44.047982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:37755 (local address 127.20.240.66:47089)
0923 17:39:44.049052 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:44.049073 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:44.049247 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:44.049599 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:44.049603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:44.049628 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:44.050022 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:44.050033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.051239 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.051245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:44.053055 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:44.053072 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.054562 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.054569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:44.054748 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:44.054773 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:39:44.054853 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 17:39:44.055028 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":8,"thread_start_us":552,"threads_started":1}
I20260923 17:39:44.055589 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:44.047982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:45191 (local address 127.20.240.66:50191)
0923 17:39:44.049205 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:44.049220 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:44.049349 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:44.049657 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:44.049661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:44.049688 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:44.050091 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:44.050101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.051596 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.051602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:44.053212 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:44.053228 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.054515 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.054522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:44.054715 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:44.054765 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 17:39:44.054853 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 17:39:44.055030 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1098,"mutex_wait_us":106,"thread_start_us":571,"threads_started":1}
W20260923 17:39:44.056669 8547 tablet.cc:2369] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:39:44.056700 8501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3102d156d1b041eba3205f07c4c85433" candidate_uuid: "da15875b91014f2f8ae59a6e558bef87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e732c84b2254e19a3653f66359f0c55" is_pre_election: true
I20260923 17:39:44.057056 8772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3102d156d1b041eba3205f07c4c85433" candidate_uuid: "da15875b91014f2f8ae59a6e558bef87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "350133d188d843ed8f21e96de5cd1ba6" is_pre_election: true
I20260923 17:39:44.057361 8501 raft_consensus.cc:2454] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da15875b91014f2f8ae59a6e558bef87 in term 0.
I20260923 17:39:44.057588 8772 raft_consensus.cc:2454] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da15875b91014f2f8ae59a6e558bef87 in term 0.
I20260923 17:39:44.058228 8572 leader_election.cc:302] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 350133d188d843ed8f21e96de5cd1ba6, da15875b91014f2f8ae59a6e558bef87; no voters:
I20260923 17:39:44.058772 8844 raft_consensus.cc:2781] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:39:44.059016 8844 raft_consensus.cc:482] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:39:44.059129 8844 raft_consensus.cc:3037] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:44.061525 8844 raft_consensus.cc:504] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da15875b91014f2f8ae59a6e558bef87" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45809 } } peers { permanent_uuid: "3e732c84b2254e19a3653f66359f0c55" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37755 } } peers { permanent_uuid: "350133d188d843ed8f21e96de5cd1ba6" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45191 } }
I20260923 17:39:44.062177 8844 leader_election.cc:288] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [CANDIDATE]: Term 1 election: Requested vote from peers 3e732c84b2254e19a3653f66359f0c55 (127.20.240.65:37755), 350133d188d843ed8f21e96de5cd1ba6 (127.20.240.67:45191)
I20260923 17:39:44.062884 8501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3102d156d1b041eba3205f07c4c85433" candidate_uuid: "da15875b91014f2f8ae59a6e558bef87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e732c84b2254e19a3653f66359f0c55"
I20260923 17:39:44.062907 8772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3102d156d1b041eba3205f07c4c85433" candidate_uuid: "da15875b91014f2f8ae59a6e558bef87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "350133d188d843ed8f21e96de5cd1ba6"
I20260923 17:39:44.063141 8501 raft_consensus.cc:3037] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:44.063136 8772 raft_consensus.cc:3037] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:44.066510 8501 raft_consensus.cc:2454] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da15875b91014f2f8ae59a6e558bef87 in term 1.
I20260923 17:39:44.066510 8772 raft_consensus.cc:2454] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da15875b91014f2f8ae59a6e558bef87 in term 1.
I20260923 17:39:44.067022 8572 leader_election.cc:302] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 350133d188d843ed8f21e96de5cd1ba6, da15875b91014f2f8ae59a6e558bef87; no voters:
I20260923 17:39:44.067394 8844 raft_consensus.cc:2781] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:44.067906 8844 raft_consensus.cc:686] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [term 1 LEADER]: Becoming Leader. State: Replica: da15875b91014f2f8ae59a6e558bef87, State: Running, Role: LEADER
I20260923 17:39:44.068524 8844 consensus_queue.cc:237] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da15875b91014f2f8ae59a6e558bef87" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45809 } } peers { permanent_uuid: "3e732c84b2254e19a3653f66359f0c55" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37755 } } peers { permanent_uuid: "350133d188d843ed8f21e96de5cd1ba6" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45191 } }
W20260923 17:39:44.074105 8684 tablet.cc:2369] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:39:44.073916 8356 catalog_manager.cc:5495] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 reported cstate change: term changed from 0 to 1, leader changed from <none> to da15875b91014f2f8ae59a6e558bef87 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "da15875b91014f2f8ae59a6e558bef87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da15875b91014f2f8ae59a6e558bef87" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e732c84b2254e19a3653f66359f0c55" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "350133d188d843ed8f21e96de5cd1ba6" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45191 } health_report { overall_health: UNKNOWN } } }
I20260923 17:39:44.109735 8831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:44.103938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.20.240.66:45809)
0923 17:39:44.104121 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:44.104125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:44.104146 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:39:44.104389 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:44.104394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:44.104495 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:44.104636 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:39:44.105093 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.105820 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.107404 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.107860 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.107961 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:44.108564 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:39:44.109188 (+ 624us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:39:44.109303 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:44.109441 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 17:39:44.109513 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260923 17:39:44.110373 8818 tablet.cc:2369] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:39:44.118526 8772 raft_consensus.cc:1261] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6 [term 1 FOLLOWER]: Refusing update from remote peer da15875b91014f2f8ae59a6e558bef87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:39:44.118526 8501 raft_consensus.cc:1261] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55 [term 1 FOLLOWER]: Refusing update from remote peer da15875b91014f2f8ae59a6e558bef87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:39:44.119692 8847 consensus_queue.cc:1035] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "350133d188d843ed8f21e96de5cd1ba6" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 45191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:39:44.120121 8844 consensus_queue.cc:1035] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e732c84b2254e19a3653f66359f0c55" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 37755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:39:44.133453 8851 mvcc.cc:205] Tried to move back new op lower bound from 7332598514138722304 to 7332598513948631040. Current Snapshot: MvccSnapshot[applied={T|T < 7332598514138722304}]
I20260923 17:39:44.140018 8852 mvcc.cc:205] Tried to move back new op lower bound from 7332598514138722304 to 7332598513948631040. Current Snapshot: MvccSnapshot[applied={T|T < 7332598514138722304}]
I20260923 17:39:44.148056 8356 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:39:44.148342 8356 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:39:44.152493 8356 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 444f233eb0964e95aeb25ca99979b663: Sending DeleteTablet for 3 replicas of tablet 3102d156d1b041eba3205f07c4c85433
I20260923 17:39:44.153949 8481 tablet_service.cc:1514] Processing DeleteTablet for tablet 3102d156d1b041eba3205f07c4c85433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:44 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:47104
I20260923 17:39:44.154026 8752 tablet_service.cc:1514] Processing DeleteTablet for tablet 3102d156d1b041eba3205f07c4c85433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:44 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:43532
I20260923 17:39:44.153981 8617 tablet_service.cc:1514] Processing DeleteTablet for tablet 3102d156d1b041eba3205f07c4c85433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:44 UTC) from {username='kudu', principal='kudu/127.20.240.126@KRBTEST.COM'} at 127.0.0.1:60758
I20260923 17:39:44.156376 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 8418
I20260923 17:39:44.169800 8861 tablet_replica.cc:330] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6: stopping tablet replica
I20260923 17:39:44.170392 8861 raft_consensus.cc:2229] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:39:44.170974 8861 raft_consensus.cc:2258] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:44.173823 8861 ts_tablet_manager.cc:1905] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:44.182025 8861 ts_tablet_manager.cc:1918] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:44.182304 8861 log.cc:1196] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/3102d156d1b041eba3205f07c4c85433
I20260923 17:39:44.182956 8861 ts_tablet_manager.cc:1939] T 3102d156d1b041eba3205f07c4c85433 P 350133d188d843ed8f21e96de5cd1ba6: Deleting consensus metadata
I20260923 17:39:44.185245 8344 catalog_manager.cc:4841] TS 350133d188d843ed8f21e96de5cd1ba6 (127.20.240.67:45191): tablet 3102d156d1b041eba3205f07c4c85433 (table test-table [id=4c456de580154f5fbfd0cefaadd00279]) successfully deleted
I20260923 17:39:44.185474 8862 tablet_replica.cc:330] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87: stopping tablet replica
I20260923 17:39:44.185971 8862 raft_consensus.cc:2229] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:39:44.186684 8862 raft_consensus.cc:2258] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:44.187618 8860 tablet_replica.cc:330] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55: stopping tablet replica
I20260923 17:39:44.188074 8860 raft_consensus.cc:2229] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:39:44.188649 8860 raft_consensus.cc:2258] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:44.190925 8860 ts_tablet_manager.cc:1905] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 17:39:44 dist-test-slave-ppjl krb5kdc[8312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.65@KRBTEST.COM
I20260923 17:39:44.198994 8860 ts_tablet_manager.cc:1918] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:44.199342 8860 log.cc:1196] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/3102d156d1b041eba3205f07c4c85433
I20260923 17:39:44.200023 8860 ts_tablet_manager.cc:1939] T 3102d156d1b041eba3205f07c4c85433 P 3e732c84b2254e19a3653f66359f0c55: Deleting consensus metadata
I20260923 17:39:44.200551 8862 ts_tablet_manager.cc:1905] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:44.202075 8343 catalog_manager.cc:4841] TS 3e732c84b2254e19a3653f66359f0c55 (127.20.240.65:37755): tablet 3102d156d1b041eba3205f07c4c85433 (table test-table [id=4c456de580154f5fbfd0cefaadd00279]) successfully deleted
I20260923 17:39:44.204342 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:44.158061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47108 (local address 127.20.240.65:37755)
0923 17:39:44.158285 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:44.158293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:44.182235 (+ 23942us) server_negotiation.cc:407] Connection header received
0923 17:39:44.183024 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:44.183030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:44.183126 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:44.183233 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:44.185204 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.186427 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.188504 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.189372 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.189513 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:44.195865 (+ 6352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:44.195886 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:44.195906 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:44.195967 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:44.200477 (+ 4510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:44.201553 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:44.201564 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:44.201567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:44.201754 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:44.202274 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:44.202279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:44.202283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:44.202760 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:44.203069 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:44.203714 (+ 645us) server_negotiation.cc:299] Negotiation successful
0923 17:39:44.203969 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 17:39:44.205878 8541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:44.206919 8862 ts_tablet_manager.cc:1918] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:44.207123 8862 log.cc:1196] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/3102d156d1b041eba3205f07c4c85433
I20260923 17:39:44.207713 8862 ts_tablet_manager.cc:1939] T 3102d156d1b041eba3205f07c4c85433 P da15875b91014f2f8ae59a6e558bef87: Deleting consensus metadata
I20260923 17:39:44.209894 8341 catalog_manager.cc:4841] TS da15875b91014f2f8ae59a6e558bef87 (127.20.240.66:45809): tablet 3102d156d1b041eba3205f07c4c85433 (table test-table [id=4c456de580154f5fbfd0cefaadd00279]) successfully deleted
I20260923 17:39:44.344307 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 8418
I20260923 17:39:44.359958 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 8552
Sep 23 17:39:44 dist-test-slave-ppjl krb5kdc[8312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.66@KRBTEST.COM
I20260923 17:39:44.378852 8831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:44.360963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.20.240.66:45809)
0923 17:39:44.361125 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:44.361129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:44.361150 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:39:44.361288 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:44.361294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:44.361380 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:44.361488 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:44.363436 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.364305 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.366181 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.366630 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.366719 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:44.372171 (+ 5452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:44.372188 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:44.372204 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:44.372249 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:44.375465 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:44.376361 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:44.376370 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:44.376372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:44.376523 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:44.376959 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:44.376963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:44.376965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:44.377376 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:44.377649 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:44.378171 (+ 522us) server_negotiation.cc:299] Negotiation successful
0923 17:39:44.378388 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:39:44.379680 8678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:44.495858 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 8552
I20260923 17:39:44.511515 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 8688
Sep 23 17:39:44 dist-test-slave-ppjl krb5kdc[8312](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.67@KRBTEST.COM
I20260923 17:39:44.527262 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:44.512426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.20.240.67:45191)
0923 17:39:44.512627 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:44.512633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:44.512661 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:39:44.512854 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:44.512858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:44.512926 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:44.513026 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:44.514323 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.515050 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.516326 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:44.516699 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:44.516817 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:44.521442 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:44.521469 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:44.521486 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:44.521529 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:44.524312 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:44.525118 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:44.525127 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:44.525129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:44.525274 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:44.525668 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:44.525672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:44.525675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:44.526056 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:44.526349 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:44.526668 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 17:39:44.526886 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:39:44.528005 8812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:44.618219 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 8688
I20260923 17:39:44.631649 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 8325
I20260923 17:39:44.632846 8387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:44.744839 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 8325
2026-09-23T17:39:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4323 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T17:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:39:44Z Disabled control of system clock
I20260923 17:39:44.818923 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:44123
--webserver_interface=127.20.240.126
--webserver_port=0
--builtin_ntp_servers=127.20.240.84:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:44123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:39:45.162992 8871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:45.163359 8871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:45.163462 8871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:45.170984 8871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:39:45.171118 8871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:45.171177 8871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:45.171260 8871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:39:45.171324 8871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:39:45.180939 8871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:44123
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:44123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:45.182283 8871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:45.184142 8871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:45.195830 8876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:45.195884 8880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:45.195861 8877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:45.196596 8871 server_base.cc:1034] running on GCE node
I20260923 17:39:45.197414 8871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:45.198941 8871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:45.200133 8871 hybrid_clock.cc:648] HybridClock initialized: now 1790185185200076 us; error 56 us; skew 500 ppm
I20260923 17:39:45.200495 8871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:39:45.202805 8871 webserver.cc:460] Webserver started at http://127.20.240.126:32843/ using document root <none> and password file <none>
I20260923 17:39:45.203666 8871 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:45.203787 8871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:45.204167 8871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:45.207240 8871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "6a8f56a88f034a4880dfcfcb1b4e3ed9"
format_stamp: "Formatted at 2026-09-23 17:39:45 on dist-test-slave-ppjl"
server_key: "afea4180e2c0e32e27eca3ba9199f188"
server_key_iv: "3c27b61143b853f8becec98610e50b4d"
server_key_version: "encryptionkey@0"
I20260923 17:39:45.208098 8871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "6a8f56a88f034a4880dfcfcb1b4e3ed9"
format_stamp: "Formatted at 2026-09-23 17:39:45 on dist-test-slave-ppjl"
server_key: "afea4180e2c0e32e27eca3ba9199f188"
server_key_iv: "3c27b61143b853f8becec98610e50b4d"
server_key_version: "encryptionkey@0"
I20260923 17:39:45.213697 8871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260923 17:39:45.217546 8886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:45.219066 8871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:39:45.219280 8871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "6a8f56a88f034a4880dfcfcb1b4e3ed9"
format_stamp: "Formatted at 2026-09-23 17:39:45 on dist-test-slave-ppjl"
server_key: "afea4180e2c0e32e27eca3ba9199f188"
server_key_iv: "3c27b61143b853f8becec98610e50b4d"
server_key_version: "encryptionkey@0"
I20260923 17:39:45.219508 8871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:45.231942 8871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:45.232784 8871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:45.233114 8871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:45.251111 8871 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:44123
I20260923 17:39:45.251129 8937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:44123 every 8 connection(s)
I20260923 17:39:45.252637 8871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:39:45.253717 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 8871
I20260923 17:39:45.253896 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:39:45.254555 21441 external_mini_cluster.cc:1428] Setting key 85c06baac8eac9040dc68990bbb3dba2
I20260923 17:39:45.257691 8938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:45.268699 8941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:45.256678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.20.240.126:44123)
0923 17:39:45.257738 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:45.257753 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:45.257809 (+ 56us) server_negotiation.cc:407] Connection header received
0923 17:39:45.259647 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:45.259672 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:45.260151 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:45.260554 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:45.261154 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:45.262587 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:45.264073 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:45.264776 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:45.264951 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:45.266122 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:45.266131 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:45.266145 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:45.266786 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:45.266878 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:45.266903 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:45.267018 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:45.267324 (+ 306us) server_negotiation.cc:299] Negotiation successful
0923 17:39:45.267655 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":370,"threads_started":1}
I20260923 17:39:45.269145 8938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9: Bootstrap starting.
I20260923 17:39:45.272516 8938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:45.273567 8938 log.cc:824] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:45.276845 8938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9: No bootstrap required, opened a new log
I20260923 17:39:45.284636 8938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8f56a88f034a4880dfcfcb1b4e3ed9" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44123 } }
I20260923 17:39:45.285041 8938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:45.285147 8938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a8f56a88f034a4880dfcfcb1b4e3ed9, State: Initialized, Role: FOLLOWER
I20260923 17:39:45.285739 8938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8f56a88f034a4880dfcfcb1b4e3ed9" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44123 } }
I20260923 17:39:45.285964 8938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:39:45.286092 8938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:39:45.286247 8938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:45.288791 8938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8f56a88f034a4880dfcfcb1b4e3ed9" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44123 } }
I20260923 17:39:45.289314 8938 leader_election.cc:302] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a8f56a88f034a4880dfcfcb1b4e3ed9; no voters:
I20260923 17:39:45.289984 8938 leader_election.cc:288] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:39:45.290129 8943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:45.290817 8943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [term 1 LEADER]: Becoming Leader. State: Replica: 6a8f56a88f034a4880dfcfcb1b4e3ed9, State: Running, Role: LEADER
I20260923 17:39:45.291373 8943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8f56a88f034a4880dfcfcb1b4e3ed9" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44123 } }
I20260923 17:39:45.292071 8938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:39:45.294109 8944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a8f56a88f034a4880dfcfcb1b4e3ed9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8f56a88f034a4880dfcfcb1b4e3ed9" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44123 } } }
I20260923 17:39:45.294576 8944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:45.294234 8945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a8f56a88f034a4880dfcfcb1b4e3ed9. Latest consensus state: current_term: 1 leader_uuid: "6a8f56a88f034a4880dfcfcb1b4e3ed9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8f56a88f034a4880dfcfcb1b4e3ed9" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44123 } } }
I20260923 17:39:45.294983 8945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9 [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:45.296195 8952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:39:45.300904 8952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:39:45.308938 8952 catalog_manager.cc:1347] Generated new cluster ID: a756d0d4f9f04e299b62530dfdc34a01
I20260923 17:39:45.309065 8952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:39:45.322196 8952 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:39:45.324302 8952 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:39:45.336522 8952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9: Generated new TSK 0
I20260923 17:39:45.337731 8952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:39:45.343288 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44123
--builtin_ntp_servers=127.20.240.84:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:39:45.682250 8962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:45.682567 8962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:45.682657 8962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:45.690342 8962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:45.690465 8962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:45.690577 8962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20260923 17:39:45.700318 8962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:45.701714 8962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:45.703573 8962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:45.717134 8967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:45.717209 8968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:45.717235 8970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:45.718327 8962 server_base.cc:1034] running on GCE node
I20260923 17:39:45.719018 8962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:45.720207 8962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:45.721467 8962 hybrid_clock.cc:648] HybridClock initialized: now 1790185185721403 us; error 50 us; skew 500 ppm
I20260923 17:39:45.721968 8962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:39:45.725199 8962 webserver.cc:460] Webserver started at http://127.20.240.65:42171/ using document root <none> and password file <none>
I20260923 17:39:45.726295 8962 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:45.726416 8962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:45.726744 8962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:45.730125 8962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/data/instance:
uuid: "6d48c40634134f0fbe27991f840d7e42"
format_stamp: "Formatted at 2026-09-23 17:39:45 on dist-test-slave-ppjl"
server_key: "0d9c4681a7e1c58a1abda1685e12a523"
server_key_iv: "76e3f01686f29274645bd862c02f3b8f"
server_key_version: "encryptionkey@0"
I20260923 17:39:45.730967 8962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance:
uuid: "6d48c40634134f0fbe27991f840d7e42"
format_stamp: "Formatted at 2026-09-23 17:39:45 on dist-test-slave-ppjl"
server_key: "0d9c4681a7e1c58a1abda1685e12a523"
server_key_iv: "76e3f01686f29274645bd862c02f3b8f"
server_key_version: "encryptionkey@0"
I20260923 17:39:45.736703 8962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 17:39:45.740577 8976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:45.742216 8962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:39:45.742400 8962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/wal
uuid: "6d48c40634134f0fbe27991f840d7e42"
format_stamp: "Formatted at 2026-09-23 17:39:45 on dist-test-slave-ppjl"
server_key: "0d9c4681a7e1c58a1abda1685e12a523"
server_key_iv: "76e3f01686f29274645bd862c02f3b8f"
server_key_version: "encryptionkey@0"
I20260923 17:39:45.742641 8962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:45.760771 8962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:45.761679 8962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:45.761998 8962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:45.763042 8962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:45.764578 8962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:45.764671 8962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:45.764791 8962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:45.764870 8962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:45.796372 8962 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:33017
I20260923 17:39:45.796422 9089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:33017 every 8 connection(s)
I20260923 17:39:45.797737 8962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/data/info.pb
I20260923 17:39:45.808009 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 8962
I20260923 17:39:45.808177 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/wal/instance
I20260923 17:39:45.808739 21441 external_mini_cluster.cc:1428] Setting key 27b66cab8dcbefa030978b4274388f09
I20260923 17:39:45.810877 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:45.800221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:59517 (local address 127.20.240.126:44123)
0923 17:39:45.800666 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:45.800671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:45.801954 (+ 1283us) server_negotiation.cc:407] Connection header received
0923 17:39:45.803872 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:45.803879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:45.803961 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:45.804072 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:45.805231 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:45.805961 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:45.807669 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:45.808090 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:45.808206 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:45.809972 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:45.809977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:45.809980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:45.810000 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:45.810044 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:45.810057 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:45.810140 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:45.810423 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 17:39:45.810527 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":236,"threads_started":1}
I20260923 17:39:45.811595 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44123
--builtin_ntp_servers=127.20.240.84:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:39:45.812036 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:45.800690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:44123 (local address 127.20.240.65:59517)
0923 17:39:45.801764 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:45.801821 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:45.803589 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:45.804340 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:45.804349 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:45.804448 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:45.805043 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:45.805061 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:45.806116 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:45.806119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:45.807512 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:45.807526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:45.808349 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:45.808358 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:45.808704 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:45.809517 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:39:45.809519 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:45.809565 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:39:45.809682 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:45.810169 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:45.810180 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:45.810187 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:39:45.810393 (+ 206us) client_negotiation.cc:240] Negotiation successful
0923 17:39:45.810828 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":278,"threads_started":1}
I20260923 17:39:45.813683 9090 heartbeater.cc:347] Connected to a master server at 127.20.240.126:44123
I20260923 17:39:45.813998 9090 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:45.814800 9090 heartbeater.cc:511] Master 127.20.240.126:44123 requested a full tablet report, sending...
I20260923 17:39:45.817257 8903 ts_manager.cc:194] Registered new tserver with Master: 6d48c40634134f0fbe27991f840d7e42 (127.20.240.65:33017)
I20260923 17:39:45.819279 8903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.65:59517
W20260923 17:39:46.152662 9094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:46.153000 9094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:46.153128 9094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:46.160672 9094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:46.160804 9094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:46.160949 9094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20260923 17:39:46.170389 9094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:46.171766 9094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:46.173589 9094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:46.186693 9099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:46.186669 9102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:46.186625 9100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:46.187577 9094 server_base.cc:1034] running on GCE node
I20260923 17:39:46.188262 9094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:46.189311 9094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:46.190560 9094 hybrid_clock.cc:648] HybridClock initialized: now 1790185186190491 us; error 56 us; skew 500 ppm
I20260923 17:39:46.190918 9094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:39:46.193507 9094 webserver.cc:460] Webserver started at http://127.20.240.66:35509/ using document root <none> and password file <none>
I20260923 17:39:46.194195 9094 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:46.194284 9094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:46.194535 9094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:46.197640 9094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/data/instance:
uuid: "74fb41c78f10473496bd5b3aacfd6f38"
format_stamp: "Formatted at 2026-09-23 17:39:46 on dist-test-slave-ppjl"
server_key: "36cd3586132bc95b5bfe6c9a458ba2be"
server_key_iv: "f7eac60863f303d5e9e9c668f5870b7e"
server_key_version: "encryptionkey@0"
I20260923 17:39:46.198457 9094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance:
uuid: "74fb41c78f10473496bd5b3aacfd6f38"
format_stamp: "Formatted at 2026-09-23 17:39:46 on dist-test-slave-ppjl"
server_key: "36cd3586132bc95b5bfe6c9a458ba2be"
server_key_iv: "f7eac60863f303d5e9e9c668f5870b7e"
server_key_version: "encryptionkey@0"
I20260923 17:39:46.203917 9094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:39:46.207798 9108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:46.209323 9094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:39:46.209508 9094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/wal
uuid: "74fb41c78f10473496bd5b3aacfd6f38"
format_stamp: "Formatted at 2026-09-23 17:39:46 on dist-test-slave-ppjl"
server_key: "36cd3586132bc95b5bfe6c9a458ba2be"
server_key_iv: "f7eac60863f303d5e9e9c668f5870b7e"
server_key_version: "encryptionkey@0"
I20260923 17:39:46.209719 9094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:46.230885 9094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:46.231758 9094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:46.232054 9094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:46.233059 9094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:46.234544 9094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:46.234632 9094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:46.234731 9094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:46.234791 9094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:46.265651 9094 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:45629
I20260923 17:39:46.265688 9221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:45629 every 8 connection(s)
I20260923 17:39:46.267289 9094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/data/info.pb
I20260923 17:39:46.269193 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 9094
I20260923 17:39:46.269358 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/wal/instance
I20260923 17:39:46.269883 21441 external_mini_cluster.cc:1428] Setting key 1ce71fac3901e37171d446b06fa18894
I20260923 17:39:46.273226 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44123
--builtin_ntp_servers=127.20.240.84:44751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:39:46.282511 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:46.270064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.66:57385 (local address 127.20.240.126:44123)
0923 17:39:46.270289 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:46.270296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:46.271999 (+ 1703us) server_negotiation.cc:407] Connection header received
0923 17:39:46.274473 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:46.274478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:46.274549 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:46.274639 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:46.276179 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.276954 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.279198 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.279689 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.279839 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:46.281546 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:46.281553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:46.281556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:46.281580 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:46.281617 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:46.281631 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:46.281714 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:46.281981 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 17:39:46.282097 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:39:46.283550 9224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:46.270741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:44123 (local address 127.20.240.66:57385)
0923 17:39:46.271838 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:46.271890 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:46.274131 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:46.274948 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:46.274962 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:46.275093 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:46.275967 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:46.276001 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.277138 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.277143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:46.278973 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:46.278990 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.279964 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.279970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:46.280267 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:46.281139 (+ 872us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:39:46.281142 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:46.281198 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:39:46.281315 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:46.281730 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:46.281740 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:46.281748 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:39:46.281947 (+ 199us) client_negotiation.cc:240] Negotiation successful
0923 17:39:46.282338 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":353,"threads_started":1}
I20260923 17:39:46.285140 9222 heartbeater.cc:347] Connected to a master server at 127.20.240.126:44123
I20260923 17:39:46.285451 9222 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:46.286242 9222 heartbeater.cc:511] Master 127.20.240.126:44123 requested a full tablet report, sending...
I20260923 17:39:46.287808 8903 ts_manager.cc:194] Registered new tserver with Master: 74fb41c78f10473496bd5b3aacfd6f38 (127.20.240.66:45629)
I20260923 17:39:46.288645 8903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.66:57385
W20260923 17:39:46.605468 9225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:46.605772 9225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:46.605899 9225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:46.613137 9225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:46.613240 9225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:46.613376 9225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20260923 17:39:46.622658 9225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:46.624154 9225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:46.625927 9225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:46.638832 9233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:46.638860 9230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:46.638968 9231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:46.640023 9225 server_base.cc:1034] running on GCE node
I20260923 17:39:46.640658 9225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:46.641666 9225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:46.642915 9225 hybrid_clock.cc:648] HybridClock initialized: now 1790185186642865 us; error 37 us; skew 500 ppm
I20260923 17:39:46.643292 9225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:39:46.645740 9225 webserver.cc:460] Webserver started at http://127.20.240.67:44593/ using document root <none> and password file <none>
I20260923 17:39:46.646535 9225 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:46.646629 9225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:46.646906 9225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:46.649955 9225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/data/instance:
uuid: "b6120085b0dc4f268217a3baa10af604"
format_stamp: "Formatted at 2026-09-23 17:39:46 on dist-test-slave-ppjl"
server_key: "276c6dce055fcc644657f2b1192869e1"
server_key_iv: "f310913b59d14063dd3ca584ea998090"
server_key_version: "encryptionkey@0"
I20260923 17:39:46.650723 9225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance:
uuid: "b6120085b0dc4f268217a3baa10af604"
format_stamp: "Formatted at 2026-09-23 17:39:46 on dist-test-slave-ppjl"
server_key: "276c6dce055fcc644657f2b1192869e1"
server_key_iv: "f310913b59d14063dd3ca584ea998090"
server_key_version: "encryptionkey@0"
I20260923 17:39:46.656180 9225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:39:46.659998 9239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:46.661507 9225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:39:46.661664 9225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/wal
uuid: "b6120085b0dc4f268217a3baa10af604"
format_stamp: "Formatted at 2026-09-23 17:39:46 on dist-test-slave-ppjl"
server_key: "276c6dce055fcc644657f2b1192869e1"
server_key_iv: "f310913b59d14063dd3ca584ea998090"
server_key_version: "encryptionkey@0"
I20260923 17:39:46.661868 9225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:46.696795 9225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:46.697693 9225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:46.697973 9225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:46.698979 9225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:39:46.700557 9225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:39:46.700649 9225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:46.700742 9225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:39:46.700803 9225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:46.734526 9225 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:44675
I20260923 17:39:46.734591 9351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:44675 every 8 connection(s)
I20260923 17:39:46.736136 9225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/data/info.pb
I20260923 17:39:46.741525 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 9225
I20260923 17:39:46.741688 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/wal/instance
I20260923 17:39:46.742139 21441 external_mini_cluster.cc:1428] Setting key 0d4647e42f75e64e6c7dd89b330243cb
I20260923 17:39:46.750491 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:46.738981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.67:34171 (local address 127.20.240.126:44123)
0923 17:39:46.739238 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:46.739245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:46.740742 (+ 1497us) server_negotiation.cc:407] Connection header received
0923 17:39:46.742755 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:46.742762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:46.742857 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:46.743081 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:46.744624 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.745648 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.747658 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.748173 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.748294 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:46.749660 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:46.749665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:46.749668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:46.749698 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:46.749741 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:46.749754 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:46.749834 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:46.750128 (+ 294us) server_negotiation.cc:299] Negotiation successful
0923 17:39:46.750213 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 17:39:46.751608 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:46.739458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:44123 (local address 127.20.240.67:34171)
0923 17:39:46.740558 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:46.740597 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:46.742410 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:46.743471 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:46.743484 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:46.743622 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:39:46.744445 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:46.744474 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.745943 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.745949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:46.747510 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:46.747525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.748399 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.748406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:46.748633 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:46.749306 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:39:46.749308 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:39:46.749351 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:39:46.749445 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:39:46.749914 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:39:46.749924 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:39:46.749931 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:39:46.750097 (+ 166us) client_negotiation.cc:240] Negotiation successful
0923 17:39:46.750428 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":413,"threads_started":1}
I20260923 17:39:46.753130 9352 heartbeater.cc:347] Connected to a master server at 127.20.240.126:44123
I20260923 17:39:46.753437 9352 heartbeater.cc:464] Registering TS with master...
I20260923 17:39:46.754215 9352 heartbeater.cc:511] Master 127.20.240.126:44123 requested a full tablet report, sending...
I20260923 17:39:46.755735 8903 ts_manager.cc:194] Registered new tserver with Master: b6120085b0dc4f268217a3baa10af604 (127.20.240.67:44675)
I20260923 17:39:46.756716 8903 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.67:34171
I20260923 17:39:46.757977 21441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:39:46.767726 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:46.761981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58168 (local address 127.20.240.126:44123)
0923 17:39:46.762164 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:46.762169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:46.762539 (+ 370us) server_negotiation.cc:407] Connection header received
0923 17:39:46.762673 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:46.762677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:46.762742 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:46.762823 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:46.763248 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.763945 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.765361 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.765800 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.765923 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:46.767059 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:46.767064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:46.767067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:46.767085 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:46.767119 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:46.767131 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:46.767247 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:46.767411 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 17:39:46.767488 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:39:46.771891 8903 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:39:46.779265 8903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:39:46.820657 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:46.794439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:44675 (local address 127.0.0.1:60538)
0923 17:39:46.810622 (+ 16183us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:46.810638 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:46.810740 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:46.812319 (+ 1579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:46.812328 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:46.812363 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:46.812736 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:46.812751 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.814714 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.814719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:46.817155 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:46.817172 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.819582 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.819589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:46.819819 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:46.819877 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 17:39:46.820016 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 17:39:46.820228 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16065,"mutex_wait_us":1,"thread_start_us":6646,"threads_started":1}
I20260923 17:39:46.820858 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:46.795023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.20.240.67:44675)
0923 17:39:46.811542 (+ 16519us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:46.811552 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:46.811591 (+ 39us) server_negotiation.cc:407] Connection header received
0923 17:39:46.811706 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:46.811714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:46.812040 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:46.812267 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:46.812981 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.814580 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.817412 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.818990 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.819125 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:46.819458 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:46.820344 (+ 886us) server_negotiation.cc:299] Negotiation successful
0923 17:39:46.820541 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16325,"thread_start_us":16542,"threads_started":1}
I20260923 17:39:46.821189 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:46.792073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:45629 (local address 127.0.0.1:58440)
0923 17:39:46.801540 (+ 9467us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:46.801594 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:46.801800 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:46.814063 (+ 12263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:46.814068 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:46.814098 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:46.814428 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:46.814439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.816524 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.816528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:46.818537 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:46.818552 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.820401 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.820407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:46.820593 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:46.820629 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:39:46.820698 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 17:39:46.820833 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9151,"mutex_wait_us":2,"thread_start_us":18279,"threads_started":1}
I20260923 17:39:46.821549 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:46.792446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.20.240.65:33017)
0923 17:39:46.812996 (+ 20550us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:46.813006 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:46.813037 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:39:46.813139 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:46.813147 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:46.813410 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:46.813617 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:46.814284 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.816070 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.818423 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.819821 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.819945 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:46.820222 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:46.821023 (+ 801us) server_negotiation.cc:299] Negotiation successful
0923 17:39:46.821205 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20331,"thread_start_us":20239,"threads_started":1}
I20260923 17:39:46.822527 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:46.792227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.20.240.66:45629)
0923 17:39:46.813407 (+ 21180us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:46.813414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:46.813445 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:39:46.813532 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:46.813538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:46.813821 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:46.814049 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:46.814837 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.816373 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.818760 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.820174 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.821681 (+ 1507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:46.821959 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:46.822029 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 17:39:46.822196 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21024,"thread_start_us":20896,"threads_started":1}
I20260923 17:39:46.823343 9090 heartbeater.cc:503] Master 127.20.240.126:44123 was elected leader, sending a full tablet report...
I20260923 17:39:46.820852 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:46.792140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.65:33017 (local address 127.0.0.1:43078)
0923 17:39:46.809925 (+ 17785us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:46.809952 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:46.810137 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:46.813622 (+ 3485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:46.813627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:46.813659 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:46.814051 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:46.814069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.816235 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.816241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:46.818175 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:46.818188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.820030 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.820036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:46.820207 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:46.820239 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:39:46.820324 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 17:39:46.820457 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17600,"mutex_wait_us":106,"thread_start_us":18680,"threads_started":1}
I20260923 17:39:46.825614 9287 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7baa9856d84dbfa9193f8e361a918a (DEFAULT_TABLE table=test-table [id=2c35aff4eaa84bda88a7789b779dac79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:46.827080 9287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7baa9856d84dbfa9193f8e361a918a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:46.826691 9156 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7baa9856d84dbfa9193f8e361a918a (DEFAULT_TABLE table=test-table [id=2c35aff4eaa84bda88a7789b779dac79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:46.827814 9156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7baa9856d84dbfa9193f8e361a918a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:46.828898 9025 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7baa9856d84dbfa9193f8e361a918a (DEFAULT_TABLE table=test-table [id=2c35aff4eaa84bda88a7789b779dac79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:39:46.830323 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7baa9856d84dbfa9193f8e361a918a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:46.840130 9370 tablet_bootstrap.cc:492] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38: Bootstrap starting.
I20260923 17:39:46.844164 9371 tablet_bootstrap.cc:492] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604: Bootstrap starting.
I20260923 17:39:46.844537 9370 tablet_bootstrap.cc:654] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:46.845813 9372 tablet_bootstrap.cc:492] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42: Bootstrap starting.
I20260923 17:39:46.846069 9370 log.cc:824] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:46.848517 9371 tablet_bootstrap.cc:654] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:46.849184 9372 tablet_bootstrap.cc:654] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:46.849749 9371 log.cc:824] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:46.850315 9372 log.cc:824] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:46.852033 9370 tablet_bootstrap.cc:492] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38: No bootstrap required, opened a new log
I20260923 17:39:46.852432 9370 ts_tablet_manager.cc:1397] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260923 17:39:46.853976 9371 tablet_bootstrap.cc:492] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604: No bootstrap required, opened a new log
I20260923 17:39:46.854305 9371 ts_tablet_manager.cc:1397] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260923 17:39:46.858086 9372 tablet_bootstrap.cc:492] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42: No bootstrap required, opened a new log
I20260923 17:39:46.858393 9372 ts_tablet_manager.cc:1397] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260923 17:39:46.860132 9370 raft_consensus.cc:348] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fb41c78f10473496bd5b3aacfd6f38" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45629 } } peers { permanent_uuid: "6d48c40634134f0fbe27991f840d7e42" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33017 } } peers { permanent_uuid: "b6120085b0dc4f268217a3baa10af604" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44675 } }
I20260923 17:39:46.860544 9370 raft_consensus.cc:374] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:46.860670 9370 raft_consensus.cc:729] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74fb41c78f10473496bd5b3aacfd6f38, State: Initialized, Role: FOLLOWER
I20260923 17:39:46.861276 9370 consensus_queue.cc:260] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fb41c78f10473496bd5b3aacfd6f38" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45629 } } peers { permanent_uuid: "6d48c40634134f0fbe27991f840d7e42" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33017 } } peers { permanent_uuid: "b6120085b0dc4f268217a3baa10af604" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44675 } }
I20260923 17:39:46.864630 9371 raft_consensus.cc:348] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fb41c78f10473496bd5b3aacfd6f38" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45629 } } peers { permanent_uuid: "6d48c40634134f0fbe27991f840d7e42" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33017 } } peers { permanent_uuid: "b6120085b0dc4f268217a3baa10af604" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44675 } }
I20260923 17:39:46.865095 9371 raft_consensus.cc:374] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:46.865190 9371 raft_consensus.cc:729] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6120085b0dc4f268217a3baa10af604, State: Initialized, Role: FOLLOWER
I20260923 17:39:46.865861 9371 consensus_queue.cc:260] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fb41c78f10473496bd5b3aacfd6f38" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45629 } } peers { permanent_uuid: "6d48c40634134f0fbe27991f840d7e42" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33017 } } peers { permanent_uuid: "b6120085b0dc4f268217a3baa10af604" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44675 } }
I20260923 17:39:46.867123 9352 heartbeater.cc:503] Master 127.20.240.126:44123 was elected leader, sending a full tablet report...
I20260923 17:39:46.867789 9371 ts_tablet_manager.cc:1428] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260923 17:39:46.868748 9372 raft_consensus.cc:348] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fb41c78f10473496bd5b3aacfd6f38" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45629 } } peers { permanent_uuid: "6d48c40634134f0fbe27991f840d7e42" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33017 } } peers { permanent_uuid: "b6120085b0dc4f268217a3baa10af604" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44675 } }
I20260923 17:39:46.869163 9372 raft_consensus.cc:374] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:46.869257 9372 raft_consensus.cc:729] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d48c40634134f0fbe27991f840d7e42, State: Initialized, Role: FOLLOWER
I20260923 17:39:46.869946 9372 consensus_queue.cc:260] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fb41c78f10473496bd5b3aacfd6f38" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45629 } } peers { permanent_uuid: "6d48c40634134f0fbe27991f840d7e42" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33017 } } peers { permanent_uuid: "b6120085b0dc4f268217a3baa10af604" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44675 } }
I20260923 17:39:46.870800 9370 ts_tablet_manager.cc:1428] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38: Time spent starting tablet: real 0.018s user 0.015s sys 0.001s
I20260923 17:39:46.871325 9222 heartbeater.cc:503] Master 127.20.240.126:44123 was elected leader, sending a full tablet report...
I20260923 17:39:46.871614 9372 ts_tablet_manager.cc:1428] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20260923 17:39:46.937378 9378 raft_consensus.cc:482] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:39:46.937739 9378 raft_consensus.cc:504] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fb41c78f10473496bd5b3aacfd6f38" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45629 } } peers { permanent_uuid: "6d48c40634134f0fbe27991f840d7e42" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33017 } } peers { permanent_uuid: "b6120085b0dc4f268217a3baa10af604" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44675 } }
I20260923 17:39:46.939608 9378 leader_election.cc:288] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74fb41c78f10473496bd5b3aacfd6f38 (127.20.240.66:45629), b6120085b0dc4f268217a3baa10af604 (127.20.240.67:44675)
I20260923 17:39:46.946617 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:46.939978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:59321 (local address 127.20.240.66:45629)
0923 17:39:46.940177 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:46.940184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:46.941103 (+ 919us) server_negotiation.cc:407] Connection header received
0923 17:39:46.941300 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:46.941304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:46.941385 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:46.941458 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:46.942119 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.942805 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.944893 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.945671 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.945820 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:46.945896 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:46.946225 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 17:39:46.946355 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:39:46.946731 9380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:46.940021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.66:45629 (local address 127.20.240.65:59321)
0923 17:39:46.941002 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:46.941016 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:46.941169 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:46.941489 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:46.941493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:46.941527 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:46.941955 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:46.941968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.942934 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.942940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:46.944699 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:46.944713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.945852 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.945858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:46.946019 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:46.946098 (+ 79us) client_negotiation.cc:769] Sending connection context
0923 17:39:46.946179 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 17:39:46.946319 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":137,"spinlock_wait_cycles":18944,"thread_start_us":642,"threads_started":1}
I20260923 17:39:46.946779 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:46.940120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.240.65:41137 (local address 127.20.240.67:44675)
0923 17:39:46.940299 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:46.940306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:46.941098 (+ 792us) server_negotiation.cc:407] Connection header received
0923 17:39:46.941301 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:46.941305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:46.941366 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:46.941439 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:39:46.942158 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.942852 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.944960 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.945826 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.946020 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:46.946120 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:46.946421 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 17:39:46.946550 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:39:46.947057 9379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:46.939968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:44675 (local address 127.20.240.65:41137)
0923 17:39:46.940971 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0923 17:39:46.940994 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:39:46.941166 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:39:46.941484 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:39:46.941488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:39:46.941525 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:39:46.942013 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:46.942022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.942994 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.942999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:46.944730 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:39:46.944743 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:39:46.946020 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:39:46.946026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:39:46.946217 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:46.946249 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:39:46.946375 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 17:39:46.946512 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":139,"spinlock_wait_cycles":32640,"thread_start_us":575,"threads_started":1}
I20260923 17:39:46.947934 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7baa9856d84dbfa9193f8e361a918a" candidate_uuid: "6d48c40634134f0fbe27991f840d7e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fb41c78f10473496bd5b3aacfd6f38" is_pre_election: true
I20260923 17:39:46.948212 9307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7baa9856d84dbfa9193f8e361a918a" candidate_uuid: "6d48c40634134f0fbe27991f840d7e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6120085b0dc4f268217a3baa10af604" is_pre_election: true
I20260923 17:39:46.948635 9177 raft_consensus.cc:2454] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d48c40634134f0fbe27991f840d7e42 in term 0.
I20260923 17:39:46.948735 9307 raft_consensus.cc:2454] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d48c40634134f0fbe27991f840d7e42 in term 0.
I20260923 17:39:46.949553 8979 leader_election.cc:302] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d48c40634134f0fbe27991f840d7e42, 74fb41c78f10473496bd5b3aacfd6f38; no voters:
I20260923 17:39:46.950034 9378 raft_consensus.cc:2781] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:39:46.950177 9378 raft_consensus.cc:482] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:39:46.950297 9378 raft_consensus.cc:3037] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:46.952611 9378 raft_consensus.cc:504] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fb41c78f10473496bd5b3aacfd6f38" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45629 } } peers { permanent_uuid: "6d48c40634134f0fbe27991f840d7e42" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33017 } } peers { permanent_uuid: "b6120085b0dc4f268217a3baa10af604" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44675 } }
I20260923 17:39:46.953285 9378 leader_election.cc:288] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [CANDIDATE]: Term 1 election: Requested vote from peers 74fb41c78f10473496bd5b3aacfd6f38 (127.20.240.66:45629), b6120085b0dc4f268217a3baa10af604 (127.20.240.67:44675)
I20260923 17:39:46.953996 9307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7baa9856d84dbfa9193f8e361a918a" candidate_uuid: "6d48c40634134f0fbe27991f840d7e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6120085b0dc4f268217a3baa10af604"
I20260923 17:39:46.954018 9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f7baa9856d84dbfa9193f8e361a918a" candidate_uuid: "6d48c40634134f0fbe27991f840d7e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74fb41c78f10473496bd5b3aacfd6f38"
I20260923 17:39:46.954227 9307 raft_consensus.cc:3037] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:46.954213 9177 raft_consensus.cc:3037] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:46.956719 9177 raft_consensus.cc:2454] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d48c40634134f0fbe27991f840d7e42 in term 1.
I20260923 17:39:46.956888 9307 raft_consensus.cc:2454] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d48c40634134f0fbe27991f840d7e42 in term 1.
I20260923 17:39:46.957250 8979 leader_election.cc:302] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d48c40634134f0fbe27991f840d7e42, 74fb41c78f10473496bd5b3aacfd6f38; no voters:
I20260923 17:39:46.957597 9378 raft_consensus.cc:2781] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:46.958156 9378 raft_consensus.cc:686] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [term 1 LEADER]: Becoming Leader. State: Replica: 6d48c40634134f0fbe27991f840d7e42, State: Running, Role: LEADER
I20260923 17:39:46.958727 9378 consensus_queue.cc:237] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fb41c78f10473496bd5b3aacfd6f38" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45629 } } peers { permanent_uuid: "6d48c40634134f0fbe27991f840d7e42" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33017 } } peers { permanent_uuid: "b6120085b0dc4f268217a3baa10af604" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44675 } }
I20260923 17:39:46.963991 8902 catalog_manager.cc:5495] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d48c40634134f0fbe27991f840d7e42 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "6d48c40634134f0fbe27991f840d7e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74fb41c78f10473496bd5b3aacfd6f38" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d48c40634134f0fbe27991f840d7e42" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6120085b0dc4f268217a3baa10af604" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44675 } health_report { overall_health: UNKNOWN } } }
W20260923 17:39:46.987955 9353 tablet.cc:2369] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:39:47.018108 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:47.012239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.20.240.65:33017)
0923 17:39:47.012435 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:47.012440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:47.012458 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:39:47.012626 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:47.012630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:47.012720 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:47.012934 (+ 214us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:39:47.013300 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:47.014103 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:47.015574 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:47.016062 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:47.016159 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:47.016879 (+ 720us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:39:47.017433 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:39:47.017553 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:47.017711 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 17:39:47.017850 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 17:39:47.019483 9223 tablet.cc:2369] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:39:47.026876 9307 raft_consensus.cc:1261] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604 [term 1 FOLLOWER]: Refusing update from remote peer 6d48c40634134f0fbe27991f840d7e42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:39:47.026876 9177 raft_consensus.cc:1261] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38 [term 1 FOLLOWER]: Refusing update from remote peer 6d48c40634134f0fbe27991f840d7e42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:39:47.027894 9378 consensus_queue.cc:1035] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6120085b0dc4f268217a3baa10af604" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:39:47.028218 9381 consensus_queue.cc:1035] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74fb41c78f10473496bd5b3aacfd6f38" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 45629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:39:47.048359 9386 mvcc.cc:205] Tried to move back new op lower bound from 7332598526051737600 to 7332598525787906048. Current Snapshot: MvccSnapshot[applied={T|T < 7332598526051737600}]
I20260923 17:39:47.065891 8902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:39:47.066205 8902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:39:47.070408 8902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a8f56a88f034a4880dfcfcb1b4e3ed9: Sending DeleteTablet for 3 replicas of tablet 1f7baa9856d84dbfa9193f8e361a918a
I20260923 17:39:47.071744 9156 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7baa9856d84dbfa9193f8e361a918a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:47 UTC) from {username='slave'} at 127.0.0.1:58440
I20260923 17:39:47.071753 9287 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7baa9856d84dbfa9193f8e361a918a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:47 UTC) from {username='slave'} at 127.0.0.1:60538
I20260923 17:39:47.071748 9025 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f7baa9856d84dbfa9193f8e361a918a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:39:47 UTC) from {username='slave'} at 127.0.0.1:43078
I20260923 17:39:47.075420 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 8962
I20260923 17:39:47.085644 9394 tablet_replica.cc:330] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604: stopping tablet replica
I20260923 17:39:47.086230 9394 raft_consensus.cc:2229] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:39:47.086817 9394 raft_consensus.cc:2258] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:47.089088 9395 tablet_replica.cc:330] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38: stopping tablet replica
I20260923 17:39:47.089439 9394 ts_tablet_manager.cc:1905] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:47.089921 9396 tablet_replica.cc:330] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42: stopping tablet replica
I20260923 17:39:47.090462 9395 raft_consensus.cc:2229] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:39:47.091377 9396 raft_consensus.cc:2229] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:39:47.091509 9395 raft_consensus.cc:2258] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:47.092244 9396 raft_consensus.cc:2258] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:39:47.093998 9395 ts_tablet_manager.cc:1905] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:47.094530 9396 ts_tablet_manager.cc:1905] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:39:47.096969 9394 ts_tablet_manager.cc:1918] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:47.097254 9394 log.cc:1196] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-2/wal/wals/1f7baa9856d84dbfa9193f8e361a918a
I20260923 17:39:47.097879 9394 ts_tablet_manager.cc:1939] T 1f7baa9856d84dbfa9193f8e361a918a P b6120085b0dc4f268217a3baa10af604: Deleting consensus metadata
I20260923 17:39:47.100086 8888 catalog_manager.cc:4841] TS b6120085b0dc4f268217a3baa10af604 (127.20.240.67:44675): tablet 1f7baa9856d84dbfa9193f8e361a918a (table test-table [id=2c35aff4eaa84bda88a7789b779dac79]) successfully deleted
I20260923 17:39:47.101421 9395 ts_tablet_manager.cc:1918] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:47.101696 9395 log.cc:1196] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-1/wal/wals/1f7baa9856d84dbfa9193f8e361a918a
I20260923 17:39:47.102114 9396 ts_tablet_manager.cc:1918] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:39:47.102288 9395 ts_tablet_manager.cc:1939] T 1f7baa9856d84dbfa9193f8e361a918a P 74fb41c78f10473496bd5b3aacfd6f38: Deleting consensus metadata
I20260923 17:39:47.102356 9396 log.cc:1196] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42: Deleting WAL directory at /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184865977475-21441-0/minicluster-data/ts-0/wal/wals/1f7baa9856d84dbfa9193f8e361a918a
I20260923 17:39:47.103350 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:47.076667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.20.240.65:33017)
0923 17:39:47.076861 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:47.076868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:47.081403 (+ 4535us) server_negotiation.cc:407] Connection header received
0923 17:39:47.082365 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:47.082371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:47.082489 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:47.082569 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:47.083819 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:47.085021 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:47.089213 (+ 4192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:47.089965 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:47.090076 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:47.101261 (+ 11185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:47.101276 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:47.101295 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:47.102186 (+ 891us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:47.102257 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:47.102294 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:47.102431 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:47.102731 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 17:39:47.102857 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:39:47.102993 9396 ts_tablet_manager.cc:1939] T 1f7baa9856d84dbfa9193f8e361a918a P 6d48c40634134f0fbe27991f840d7e42: Deleting consensus metadata
I20260923 17:39:47.104084 9085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:47.105130 8889 catalog_manager.cc:4841] TS 74fb41c78f10473496bd5b3aacfd6f38 (127.20.240.66:45629): tablet 1f7baa9856d84dbfa9193f8e361a918a (table test-table [id=2c35aff4eaa84bda88a7789b779dac79]) successfully deleted
I20260923 17:39:47.203326 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 8962
I20260923 17:39:47.203514 8890 catalog_manager.cc:4841] TS 6d48c40634134f0fbe27991f840d7e42 (127.20.240.65:33017): tablet 1f7baa9856d84dbfa9193f8e361a918a (table test-table [id=2c35aff4eaa84bda88a7789b779dac79]) successfully deleted
I20260923 17:39:47.218861 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 9094
I20260923 17:39:47.226104 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:47.219741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58442 (local address 127.20.240.66:45629)
0923 17:39:47.219968 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:47.219973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:47.219991 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:39:47.220048 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:47.220051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:47.220140 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:47.220226 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:47.220720 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:47.221523 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:47.222951 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:47.223438 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:47.223546 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:47.224568 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:47.224579 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:47.224594 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:47.225299 (+ 705us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:47.225359 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:47.225383 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:47.225487 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:47.225644 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 17:39:47.225795 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:39:47.226898 9217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:47.317334 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 9094
I20260923 17:39:47.331348 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 9225
I20260923 17:39:47.337982 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:47.332165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.20.240.67:44675)
0923 17:39:47.332327 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:47.332331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:47.332351 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:39:47.332407 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:47.332411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:47.332513 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:47.332595 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:47.333003 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:47.333775 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:47.335000 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:47.335547 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:47.335640 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:47.336664 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:47.336673 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:47.336688 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:47.337262 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:47.337313 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:47.337333 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:47.337436 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:47.337616 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 17:39:47.337720 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:39:47.338651 9347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:47.428027 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 9225
I20260923 17:39:47.442051 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 8871
I20260923 17:39:47.443480 8933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:47.553799 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 8871
2026-09-23T17:39:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2808 ms)
[----------] 6 tests from EncryptionPolicyTest (21119 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:39:47 dist-test-slave-ppjl krb5kdc[9405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:39:47 dist-test-slave-ppjl krb5kdc[9405](info): set up 2 sockets
Sep 23 17:39:47 dist-test-slave-ppjl krb5kdc[9405](info): commencing operation
krb5kdc: starting...
W20260923 17:39:48.596347 21441 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:39:48 dist-test-slave-ppjl krb5kdc[9405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:39:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:39:48.855608 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:49.199827 9418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:49.200141 9418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:49.200235 9418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:49.207831 9418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:39:49.207958 9418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:49.208009 9418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:49.208071 9418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:39:49.208114 9418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:39:49.218178 9418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:49.219559 9418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:49.220854 9418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:49.232877 9423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:49.232849 9426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:49.232921 9424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:49.233804 9418 server_base.cc:1034] running on GCE node
I20260923 17:39:49.234628 9418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:49.236184 9418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:49.237432 9418 hybrid_clock.cc:648] HybridClock initialized: now 1790185189237375 us; error 41 us; skew 500 ppm
I20260923 17:39:49.237833 9418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:49 dist-test-slave-ppjl krb5kdc[9405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:49.241065 9418 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:39:49.242550 9418 webserver.cc:460] Webserver started at http://127.0.0.1:39503/ using document root <none> and password file <none>
I20260923 17:39:49.243287 9418 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:49.243419 9418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:49.243705 9418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:49.246759 9418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "55ef07e5426b4f27a02bb190f9ae6b5e"
format_stamp: "Formatted at 2026-09-23 17:39:49 on dist-test-slave-ppjl"
server_key: "3c0fe1139060ef64a63658f48718d36c"
server_key_iv: "315bb18201df0b35508d2366f6982ebe"
server_key_version: "encryptionkey@0"
I20260923 17:39:49.247581 9418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "55ef07e5426b4f27a02bb190f9ae6b5e"
format_stamp: "Formatted at 2026-09-23 17:39:49 on dist-test-slave-ppjl"
server_key: "3c0fe1139060ef64a63658f48718d36c"
server_key_iv: "315bb18201df0b35508d2366f6982ebe"
server_key_version: "encryptionkey@0"
I20260923 17:39:49.253060 9418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:39:49.257046 9434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:49.258620 9418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:39:49.258834 9418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "55ef07e5426b4f27a02bb190f9ae6b5e"
format_stamp: "Formatted at 2026-09-23 17:39:49 on dist-test-slave-ppjl"
server_key: "3c0fe1139060ef64a63658f48718d36c"
server_key_iv: "315bb18201df0b35508d2366f6982ebe"
server_key_version: "encryptionkey@0"
I20260923 17:39:49.259047 9418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:49.290083 9418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:49.294888 9418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:49.295235 9418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:49.314227 9418 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34343
I20260923 17:39:49.314275 9485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34343 every 8 connection(s)
I20260923 17:39:49.315666 9418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:39:49.320011 9486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:49.321678 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 9418
I20260923 17:39:49.321944 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:39:49.322598 21441 external_mini_cluster.cc:1428] Setting key 1625cb39ba4ac54e8c1c72dead32f946
I20260923 17:39:49.333144 9486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e: Bootstrap starting.
I20260923 17:39:49.337463 9486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:49.338972 9486 log.cc:824] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:39:49 dist-test-slave-ppjl krb5kdc[9405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:39:49.342697 9486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e: No bootstrap required, opened a new log
I20260923 17:39:49.349570 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:49.324953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58659 (local address 127.0.0.1:34343)
0923 17:39:49.326006 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:49.326020 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:49.326069 (+ 49us) server_negotiation.cc:407] Connection header received
0923 17:39:49.327877 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:49.327906 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:49.328355 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:49.328721 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:49.330577 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:49.332054 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:49.333945 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:49.334932 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:49.335120 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:49.340849 (+ 5729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:49.340863 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:49.340882 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:49.340930 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:49.344754 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:49.345803 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:49.345811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:49.345819 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:49.346013 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:49.346545 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:49.346548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:49.346551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:49.347207 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:49.347571 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:49.348083 (+ 512us) server_negotiation.cc:299] Negotiation successful
0923 17:39:49.348426 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":568,"threads_started":1}
I20260923 17:39:49.351074 9486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ef07e5426b4f27a02bb190f9ae6b5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34343 } }
I20260923 17:39:49.351537 9486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:49.351644 9486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ef07e5426b4f27a02bb190f9ae6b5e, State: Initialized, Role: FOLLOWER
I20260923 17:39:49.352221 9486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ef07e5426b4f27a02bb190f9ae6b5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34343 } }
I20260923 17:39:49.352530 9486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:39:49.352662 9486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:39:49.352824 9486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:49.354370 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:39:49.355496 9486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ef07e5426b4f27a02bb190f9ae6b5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34343 } }
I20260923 17:39:49.356016 9486 leader_election.cc:302] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55ef07e5426b4f27a02bb190f9ae6b5e; no voters:
I20260923 17:39:49.356644 9486 leader_election.cc:288] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:39:49.356812 9491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:49.358691 9486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:39:49.359468 9491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [term 1 LEADER]: Becoming Leader. State: Replica: 55ef07e5426b4f27a02bb190f9ae6b5e, State: Running, Role: LEADER
I20260923 17:39:49.360226 9491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ef07e5426b4f27a02bb190f9ae6b5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34343 } }
I20260923 17:39:49.363481 9494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55ef07e5426b4f27a02bb190f9ae6b5e. Latest consensus state: current_term: 1 leader_uuid: "55ef07e5426b4f27a02bb190f9ae6b5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ef07e5426b4f27a02bb190f9ae6b5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34343 } } }
I20260923 17:39:49.363929 9494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:49.364698 9492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55ef07e5426b4f27a02bb190f9ae6b5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55ef07e5426b4f27a02bb190f9ae6b5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34343 } } }
I20260923 17:39:49.365012 9492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:49.369310 9500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:39:49.375339 9500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:39:49.389430 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:49.367939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60191 (local address 127.0.0.1:34343)
0923 17:39:49.368139 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:49.368145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:49.368717 (+ 572us) server_negotiation.cc:407] Connection header received
0923 17:39:49.368936 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:49.368943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:49.369050 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:49.369182 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:49.371208 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:49.372719 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:49.374839 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:49.375756 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:49.375912 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:49.379846 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:49.379854 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:49.379860 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:49.379915 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:49.384213 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:49.385539 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:49.385545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:49.385550 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:49.385730 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:49.386345 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:49.386351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:49.386355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:49.386839 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:49.387127 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:49.388715 (+ 1588us) server_negotiation.cc:299] Negotiation successful
0923 17:39:49.388954 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:39:49.398173 9500 catalog_manager.cc:1347] Generated new cluster ID: b7b69bf4d2f34e53b31571dd64498570
I20260923 17:39:49.398312 9500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:39:49.428555 9500 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:39:49.429986 9500 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:39:49.446812 9500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55ef07e5426b4f27a02bb190f9ae6b5e: Generated new TSK 0
I20260923 17:39:49.447777 9500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:39:49.463433 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 9418
I20260923 17:39:49.465741 9481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:49.580492 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 9418
W20260923 17:39:49.595893 9403 connection.cc:535] client connection to 127.0.0.1:34343 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34343): Connection reset by peer (error 104)
2026-09-23T17:39:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2022 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:39:49 dist-test-slave-ppjl krb5kdc[9519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:39:49 dist-test-slave-ppjl krb5kdc[9519](info): set up 2 sockets
Sep 23 17:39:49 dist-test-slave-ppjl krb5kdc[9519](info): commencing operation
krb5kdc: starting...
W20260923 17:39:50.616086 21441 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:39:50 dist-test-slave-ppjl krb5kdc[9519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:39:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:39:50.867131 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:51.211699 9532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:51.212191 9532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:51.212393 9532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:51.220274 9532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:39:51.220407 9532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:51.220484 9532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:51.220566 9532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:39:51.220625 9532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:39:51.230819 9532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35735
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:51.232169 9532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:51.233512 9532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:51.245344 9537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:51.245283 9538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:51.245311 9540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:51.246059 9532 server_base.cc:1034] running on GCE node
I20260923 17:39:51.246848 9532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:51.248394 9532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:51.249605 9532 hybrid_clock.cc:648] HybridClock initialized: now 1790185191249555 us; error 37 us; skew 500 ppm
I20260923 17:39:51.250029 9532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:51 dist-test-slave-ppjl krb5kdc[9519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:51.253371 9532 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:39:51.255060 9532 webserver.cc:460] Webserver started at http://127.0.0.1:34677/ using document root <none> and password file <none>
I20260923 17:39:51.255800 9532 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:51.255925 9532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:51.256220 9532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:51.259382 9532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "3d29ba9558e84955b8243ee435f54c21"
format_stamp: "Formatted at 2026-09-23 17:39:51 on dist-test-slave-ppjl"
server_key: "bcc6a26aa21d3d5eeb6886e0a3803557"
server_key_iv: "24c34e412adebb029878e850f28407fc"
server_key_version: "encryptionkey@0"
I20260923 17:39:51.260175 9532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "3d29ba9558e84955b8243ee435f54c21"
format_stamp: "Formatted at 2026-09-23 17:39:51 on dist-test-slave-ppjl"
server_key: "bcc6a26aa21d3d5eeb6886e0a3803557"
server_key_iv: "24c34e412adebb029878e850f28407fc"
server_key_version: "encryptionkey@0"
I20260923 17:39:51.265996 9532 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 17:39:51.269961 9547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:51.271499 9532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:39:51.271684 9532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "3d29ba9558e84955b8243ee435f54c21"
format_stamp: "Formatted at 2026-09-23 17:39:51 on dist-test-slave-ppjl"
server_key: "bcc6a26aa21d3d5eeb6886e0a3803557"
server_key_iv: "24c34e412adebb029878e850f28407fc"
server_key_version: "encryptionkey@0"
I20260923 17:39:51.271888 9532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:51.284449 9532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:51.288746 9532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:51.289013 9532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:51.307615 9532 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35735
I20260923 17:39:51.307648 9599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35735 every 8 connection(s)
I20260923 17:39:51.309052 9532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:39:51.312052 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 9532
I20260923 17:39:51.312223 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:39:51.312641 21441 external_mini_cluster.cc:1428] Setting key 96ec884088371774c142acca89aa1f7d
I20260923 17:39:51.312979 9600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:51.328069 9600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21: Bootstrap starting.
I20260923 17:39:51.328234 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:51.314392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37333 (local address 127.0.0.1:35735)
0923 17:39:51.315284 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:51.315300 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:51.315353 (+ 53us) server_negotiation.cc:407] Connection header received
0923 17:39:51.317757 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:51.317788 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:51.318339 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:51.318841 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:51.319432 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:51.321431 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:51.322927 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:51.324015 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:51.324201 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:51.325353 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:51.325364 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:51.325388 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:51.326128 (+ 740us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:51.326226 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:51.326257 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:51.326421 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:51.326671 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 17:39:51.326858 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":337,"threads_started":1}
I20260923 17:39:51.331512 9600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:51.332376 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:39:51.332698 9600 log.cc:824] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:51.336140 9600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21: No bootstrap required, opened a new log
I20260923 17:39:51.344290 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:51.337331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58157 (local address 127.0.0.1:35735)
0923 17:39:51.337535 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:51.337541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:51.337904 (+ 363us) server_negotiation.cc:407] Connection header received
0923 17:39:51.338092 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:51.338096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:51.338165 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:51.338253 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:51.338796 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:51.339655 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:51.341614 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:51.342094 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:51.342196 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:51.343540 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:51.343545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:51.343548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:51.343567 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:51.343612 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:51.343628 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:51.343711 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:51.343903 (+ 192us) server_negotiation.cc:299] Negotiation successful
0923 17:39:51.343948 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:39:51.345741 9600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d29ba9558e84955b8243ee435f54c21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } }
I20260923 17:39:51.346127 9600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:51.346246 9600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d29ba9558e84955b8243ee435f54c21, State: Initialized, Role: FOLLOWER
I20260923 17:39:51.346791 9600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d29ba9558e84955b8243ee435f54c21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } }
I20260923 17:39:51.347003 9600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:39:51.347127 9600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:39:51.347326 9600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:51.349488 9600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d29ba9558e84955b8243ee435f54c21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } }
I20260923 17:39:51.349992 9600 leader_election.cc:302] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d29ba9558e84955b8243ee435f54c21; no voters:
I20260923 17:39:51.350637 9600 leader_election.cc:288] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:39:51.350777 9610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:51.351969 9610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [term 1 LEADER]: Becoming Leader. State: Replica: 3d29ba9558e84955b8243ee435f54c21, State: Running, Role: LEADER
I20260923 17:39:51.352542 9610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d29ba9558e84955b8243ee435f54c21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } }
I20260923 17:39:51.353045 9600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:39:51.355662 9612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d29ba9558e84955b8243ee435f54c21. Latest consensus state: current_term: 1 leader_uuid: "3d29ba9558e84955b8243ee435f54c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d29ba9558e84955b8243ee435f54c21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } } }
I20260923 17:39:51.355772 9611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d29ba9558e84955b8243ee435f54c21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d29ba9558e84955b8243ee435f54c21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } } }
I20260923 17:39:51.356079 9612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:51.356130 9611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21 [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:51.365151 9618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:39:51.370929 9618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:39:51.381011 9618 catalog_manager.cc:1347] Generated new cluster ID: cd8447bb7d6c4d61a99f997f03545e9a
I20260923 17:39:51.381141 9618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:39:51.395762 9618 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:39:51.397811 9618 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:39:51.416531 9618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d29ba9558e84955b8243ee435f54c21: Generated new TSK 0
I20260923 17:39:51.417692 9618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:39:51.425439 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 9532
W20260923 17:39:51.426692 9595 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37333
W20260923 17:39:51.427501 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 9532: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:39:51.427657 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 9532
W20260923 17:39:51.442268 9517 connection.cc:535] client connection to 127.0.0.1:35735 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35735): Connection reset by peer (error 104)
2026-09-23T17:39:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1846 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:39:51 dist-test-slave-ppjl krb5kdc[9631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:39:51 dist-test-slave-ppjl krb5kdc[9631](info): set up 2 sockets
Sep 23 17:39:51 dist-test-slave-ppjl krb5kdc[9631](info): commencing operation
krb5kdc: starting...
W20260923 17:39:52.452026 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:39:52 dist-test-slave-ppjl krb5kdc[9631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:39:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:39:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:39:52.711719 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:53.047148 9644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:53.047550 9644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:53.047721 9644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:53.055428 9644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:39:53.055547 9644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:53.055624 9644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:53.055712 9644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:39:53.055773 9644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:39:53.067073 9644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35875
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:53.068485 9644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:53.069885 9644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:53.082361 9649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:53.082453 9650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:53.082470 9652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:53.083232 9644 server_base.cc:1034] running on GCE node
I20260923 17:39:53.084040 9644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:53.085554 9644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:53.086773 9644 hybrid_clock.cc:648] HybridClock initialized: now 1790185193086705 us; error 54 us; skew 500 ppm
I20260923 17:39:53.087126 9644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:53 dist-test-slave-ppjl krb5kdc[9631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:53.096407 9644 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:39:53.098178 9644 webserver.cc:460] Webserver started at http://127.0.0.1:46553/ using document root <none> and password file <none>
I20260923 17:39:53.098922 9644 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:53.099037 9644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:53.099392 9644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:53.102501 9644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "3414b926f0804c7cb3621cce353f188e"
format_stamp: "Formatted at 2026-09-23 17:39:53 on dist-test-slave-ppjl"
server_key: "9629e4f95afea4b94293eb1e96c2b1e4"
server_key_iv: "ee66fb26978dcc0a13e33645efb1ab1e"
server_key_version: "encryptionkey@0"
I20260923 17:39:53.103355 9644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "3414b926f0804c7cb3621cce353f188e"
format_stamp: "Formatted at 2026-09-23 17:39:53 on dist-test-slave-ppjl"
server_key: "9629e4f95afea4b94293eb1e96c2b1e4"
server_key_iv: "ee66fb26978dcc0a13e33645efb1ab1e"
server_key_version: "encryptionkey@0"
I20260923 17:39:53.108894 9644 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:39:53.112711 9659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:53.114547 9644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:39:53.114720 9644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "3414b926f0804c7cb3621cce353f188e"
format_stamp: "Formatted at 2026-09-23 17:39:53 on dist-test-slave-ppjl"
server_key: "9629e4f95afea4b94293eb1e96c2b1e4"
server_key_iv: "ee66fb26978dcc0a13e33645efb1ab1e"
server_key_version: "encryptionkey@0"
I20260923 17:39:53.114912 9644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:53.120530 9644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:53.124475 9644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:53.124742 9644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:53.142992 9644 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35875
I20260923 17:39:53.143028 9710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35875 every 8 connection(s)
I20260923 17:39:53.144414 9644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:39:53.146287 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 9644
I20260923 17:39:53.146441 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:39:53.146957 21441 external_mini_cluster.cc:1428] Setting key bc03ced370d48e9368b9c134bce89bce
I20260923 17:39:53.148675 9711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:53.154974 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:53.148804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42069 (local address 127.0.0.1:35875)
0923 17:39:53.149602 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:53.149614 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:53.149655 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:39:53.151306 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:53.151329 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:53.151692 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:53.152038 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:53.153282 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:53.153289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:53.153303 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:53.153319 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:53.153414 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:53.153443 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:53.153630 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:53.153728 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 17:39:53.153988 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":332,"threads_started":1}
I20260923 17:39:53.158589 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:39:53.161737 9711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e: Bootstrap starting.
I20260923 17:39:53.165642 9711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:53.166502 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:53.163789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54341 (local address 127.0.0.1:35875)
0923 17:39:53.164001 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:53.164009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:53.164420 (+ 411us) server_negotiation.cc:407] Connection header received
0923 17:39:53.164626 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:53.164632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:53.164755 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:53.164851 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:53.165855 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:53.165859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:53.165863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:53.165879 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:53.165926 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:53.165941 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:53.166041 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:53.166196 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 17:39:53.166244 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:39:53.166982 9711 log.cc:824] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:53.169968 9711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e: No bootstrap required, opened a new log
I20260923 17:39:53.178066 9711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3414b926f0804c7cb3621cce353f188e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35875 } }
I20260923 17:39:53.178649 9711 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:53.178791 9711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3414b926f0804c7cb3621cce353f188e, State: Initialized, Role: FOLLOWER
I20260923 17:39:53.179518 9711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3414b926f0804c7cb3621cce353f188e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35875 } }
I20260923 17:39:53.179782 9711 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:39:53.179939 9711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:39:53.180122 9711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:53.182507 9711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3414b926f0804c7cb3621cce353f188e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35875 } }
I20260923 17:39:53.183073 9711 leader_election.cc:302] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3414b926f0804c7cb3621cce353f188e; no voters:
I20260923 17:39:53.183748 9711 leader_election.cc:288] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:39:53.183940 9722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:53.184577 9722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [term 1 LEADER]: Becoming Leader. State: Replica: 3414b926f0804c7cb3621cce353f188e, State: Running, Role: LEADER
I20260923 17:39:53.185154 9722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3414b926f0804c7cb3621cce353f188e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35875 } }
I20260923 17:39:53.185895 9711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:39:53.187796 9723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3414b926f0804c7cb3621cce353f188e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3414b926f0804c7cb3621cce353f188e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35875 } } }
I20260923 17:39:53.188246 9723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:53.189003 9724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3414b926f0804c7cb3621cce353f188e. Latest consensus state: current_term: 1 leader_uuid: "3414b926f0804c7cb3621cce353f188e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3414b926f0804c7cb3621cce353f188e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35875 } } }
I20260923 17:39:53.189409 9724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:53.191488 9731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:39:53.196898 9731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:39:53.207424 9731 catalog_manager.cc:1347] Generated new cluster ID: daa7fb2c600a45c78253141c9c964133
I20260923 17:39:53.207587 9731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:39:53.242345 9731 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:39:53.242956 9731 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:39:53.261919 9731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3414b926f0804c7cb3621cce353f188e: Generated new TSK 0
I20260923 17:39:53.262878 9731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:39:53.278941 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 9644
W20260923 17:39:53.280781 9706 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42069
W20260923 17:39:53.281543 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 9644: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:39:53.281703 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 9644
2026-09-23T17:39:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1854 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:39:53 dist-test-slave-ppjl krb5kdc[9743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:39:53 dist-test-slave-ppjl krb5kdc[9743](info): set up 2 sockets
Sep 23 17:39:53 dist-test-slave-ppjl krb5kdc[9743](info): commencing operation
krb5kdc: starting...
W20260923 17:39:54.308086 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:39:54 dist-test-slave-ppjl krb5kdc[9743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:39:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:39:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:39:54.562899 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:54.901716 9756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:54.902037 9756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:54.902163 9756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:54.909912 9756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:39:54.910066 9756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:54.910157 9756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:54.910244 9756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:39:54.910307 9756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:39:54.921067 9756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:54.922443 9756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:54.923902 9756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:54.936282 9761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:54.936338 9764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:54.936309 9762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:54.937196 9756 server_base.cc:1034] running on GCE node
I20260923 17:39:54.938007 9756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:54.939942 9756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:54.941217 9756 hybrid_clock.cc:648] HybridClock initialized: now 1790185194941099 us; error 80 us; skew 500 ppm
I20260923 17:39:54.941594 9756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:54 dist-test-slave-ppjl krb5kdc[9743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:54.945472 9756 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:39:54.947404 9756 webserver.cc:460] Webserver started at http://127.0.0.1:42843/ using document root <none> and password file <none>
I20260923 17:39:54.948436 9756 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:54.948570 9756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:54.948912 9756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:54.952296 9756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "c83417937cbb4c9bbfc782cf7f5fb42f"
format_stamp: "Formatted at 2026-09-23 17:39:54 on dist-test-slave-ppjl"
server_key: "4dba0b3eb6b65dc56e4186971271d7ba"
server_key_iv: "1afd0d62fdc3518d341fa1db36c30878"
server_key_version: "encryptionkey@0"
I20260923 17:39:54.953171 9756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "c83417937cbb4c9bbfc782cf7f5fb42f"
format_stamp: "Formatted at 2026-09-23 17:39:54 on dist-test-slave-ppjl"
server_key: "4dba0b3eb6b65dc56e4186971271d7ba"
server_key_iv: "1afd0d62fdc3518d341fa1db36c30878"
server_key_version: "encryptionkey@0"
I20260923 17:39:54.959043 9756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:39:54.963483 9771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:54.965252 9756 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:39:54.965523 9756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "c83417937cbb4c9bbfc782cf7f5fb42f"
format_stamp: "Formatted at 2026-09-23 17:39:54 on dist-test-slave-ppjl"
server_key: "4dba0b3eb6b65dc56e4186971271d7ba"
server_key_iv: "1afd0d62fdc3518d341fa1db36c30878"
server_key_version: "encryptionkey@0"
I20260923 17:39:54.965776 9756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:54.987041 9756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:54.992156 9756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:54.992460 9756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:55.010804 9756 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42329
I20260923 17:39:55.010823 9822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42329 every 8 connection(s)
I20260923 17:39:55.012253 9756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:39:55.016270 9823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:55.018855 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 9756
I20260923 17:39:55.019019 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:39:55.019479 21441 external_mini_cluster.cc:1428] Setting key 679021149c9c77ef446bacbd385bfd90
I20260923 17:39:55.027936 9823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f: Bootstrap starting.
I20260923 17:39:55.031080 9823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:55.032143 9823 log.cc:824] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:39:55 dist-test-slave-ppjl krb5kdc[9743](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:39:55.035002 9823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f: No bootstrap required, opened a new log
I20260923 17:39:55.042054 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:55.021399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38311 (local address 127.0.0.1:42329)
0923 17:39:55.022197 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:55.022213 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:55.022254 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:39:55.023955 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:55.023980 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:55.024445 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:55.024820 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:55.026309 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:55.027668 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:55.028938 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:55.029659 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:55.029840 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:55.034204 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:55.034213 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:55.034229 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:55.034277 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:55.037646 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:55.038442 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:55.038449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:55.038460 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:55.038644 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:55.039055 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:55.039059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:55.039061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:55.039712 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:55.040022 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:55.040474 (+ 452us) server_negotiation.cc:299] Negotiation successful
0923 17:39:55.040924 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":344,"threads_started":1}
I20260923 17:39:55.042943 9823 raft_consensus.cc:348] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83417937cbb4c9bbfc782cf7f5fb42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260923 17:39:55.043412 9823 raft_consensus.cc:374] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:55.043514 9823 raft_consensus.cc:729] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c83417937cbb4c9bbfc782cf7f5fb42f, State: Initialized, Role: FOLLOWER
I20260923 17:39:55.044135 9823 consensus_queue.cc:260] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83417937cbb4c9bbfc782cf7f5fb42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260923 17:39:55.044402 9823 raft_consensus.cc:388] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:39:55.044533 9823 raft_consensus.cc:482] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:39:55.044703 9823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:55.046306 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:39:55.047341 9823 raft_consensus.cc:504] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83417937cbb4c9bbfc782cf7f5fb42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260923 17:39:55.047865 9823 leader_election.cc:302] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c83417937cbb4c9bbfc782cf7f5fb42f; no voters:
I20260923 17:39:55.048494 9823 leader_election.cc:288] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:39:55.048700 9828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:55.049927 9828 raft_consensus.cc:686] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [term 1 LEADER]: Becoming Leader. State: Replica: c83417937cbb4c9bbfc782cf7f5fb42f, State: Running, Role: LEADER
I20260923 17:39:55.050945 9823 sys_catalog.cc:564] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:39:55.050789 9828 consensus_queue.cc:237] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83417937cbb4c9bbfc782cf7f5fb42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260923 17:39:55.054111 9831 sys_catalog.cc:455] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c83417937cbb4c9bbfc782cf7f5fb42f. Latest consensus state: current_term: 1 leader_uuid: "c83417937cbb4c9bbfc782cf7f5fb42f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83417937cbb4c9bbfc782cf7f5fb42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } } }
I20260923 17:39:55.054607 9831 sys_catalog.cc:458] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:55.055290 9829 sys_catalog.cc:455] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c83417937cbb4c9bbfc782cf7f5fb42f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83417937cbb4c9bbfc782cf7f5fb42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } } }
I20260923 17:39:55.055612 9829 sys_catalog.cc:458] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:55.070746 9836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:39:55.077888 9836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:39:55.088464 9836 catalog_manager.cc:1347] Generated new cluster ID: 3402d2ed24b347879c61064bc13e2b06
I20260923 17:39:55.088650 9836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:39:55.092300 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:55.073375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50937 (local address 127.0.0.1:42329)
0923 17:39:55.073575 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:55.073582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:55.074430 (+ 848us) server_negotiation.cc:407] Connection header received
0923 17:39:55.074657 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:55.074663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:55.074756 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:55.074883 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:55.076819 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:55.078005 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:55.080700 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:55.081464 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:55.081641 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:55.084746 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:55.084754 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:55.084759 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:39:55.084804 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:55.089090 (+ 4286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:55.089963 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:55.089973 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:55.089975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:55.090114 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:39:55.090568 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:39:55.090571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:39:55.090573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:39:55.090929 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:55.091218 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:55.091627 (+ 409us) server_negotiation.cc:299] Negotiation successful
0923 17:39:55.091862 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:39:55.110723 9836 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:39:55.112731 9836 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:39:55.121228 9836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c83417937cbb4c9bbfc782cf7f5fb42f: Generated new TSK 0
I20260923 17:39:55.122421 9836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:39:55.130023 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 9756
I20260923 17:39:55.131150 9818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:39:55.224356 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 9756
2026-09-23T17:39:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1942 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:39:55 dist-test-slave-ppjl krb5kdc[9856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:39:55 dist-test-slave-ppjl krb5kdc[9856](info): set up 2 sockets
Sep 23 17:39:55 dist-test-slave-ppjl krb5kdc[9856](info): commencing operation
krb5kdc: starting...
W20260923 17:39:56.252236 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:39:56 dist-test-slave-ppjl krb5kdc[9856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:39:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:39:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:39:56.504196 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34707
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:56.837623 9870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:56.837947 9870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:56.838073 9870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:56.845944 9870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:39:56.846081 9870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:56.846161 9870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:56.846244 9870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:39:56.846309 9870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:39:56.856436 9870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:56.857844 9870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:56.859278 9870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:56.871109 9875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:56.871148 9878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:56.871239 9876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:56.871940 9870 server_base.cc:1034] running on GCE node
I20260923 17:39:56.872689 9870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:56.874109 9870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:56.875308 9870 hybrid_clock.cc:648] HybridClock initialized: now 1790185196875251 us; error 50 us; skew 500 ppm
I20260923 17:39:56.875685 9870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:56 dist-test-slave-ppjl krb5kdc[9856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:56.878943 9870 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:39:56.880522 9870 webserver.cc:460] Webserver started at http://127.0.0.1:42111/ using document root <none> and password file <none>
I20260923 17:39:56.881217 9870 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:56.881346 9870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:56.881644 9870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:56.884744 9870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "fd12601081d74313993886449a9996e2"
format_stamp: "Formatted at 2026-09-23 17:39:56 on dist-test-slave-ppjl"
server_key: "896bcdd3cbab114322c4f46c62e111f8"
server_key_iv: "6c926656598162abea91bc8f7af6cdf0"
server_key_version: "encryptionkey@0"
I20260923 17:39:56.885828 9870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "fd12601081d74313993886449a9996e2"
format_stamp: "Formatted at 2026-09-23 17:39:56 on dist-test-slave-ppjl"
server_key: "896bcdd3cbab114322c4f46c62e111f8"
server_key_iv: "6c926656598162abea91bc8f7af6cdf0"
server_key_version: "encryptionkey@0"
I20260923 17:39:56.891762 9870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 17:39:56.895565 9885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:56.897126 9870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:39:56.897272 9870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "fd12601081d74313993886449a9996e2"
format_stamp: "Formatted at 2026-09-23 17:39:56 on dist-test-slave-ppjl"
server_key: "896bcdd3cbab114322c4f46c62e111f8"
server_key_iv: "6c926656598162abea91bc8f7af6cdf0"
server_key_version: "encryptionkey@0"
I20260923 17:39:56.897444 9870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:56.912744 9870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:56.917069 9870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:56.917305 9870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:56.935477 9870 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34707
I20260923 17:39:56.935498 9936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34707 every 8 connection(s)
I20260923 17:39:56.936920 9870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:39:56.938591 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 9870
I20260923 17:39:56.938758 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:39:56.939230 21441 external_mini_cluster.cc:1428] Setting key a341e7f9e1813b6908eede4648cb3bd2
I20260923 17:39:56.941229 9937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:56.954870 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:56.941169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46727 (local address 127.0.0.1:34707)
0923 17:39:56.941963 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:56.941977 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:56.942028 (+ 51us) server_negotiation.cc:407] Connection header received
0923 17:39:56.944204 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:56.944233 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:56.944803 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:56.945287 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:56.945837 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:56.947936 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:56.949393 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:56.950452 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:56.950634 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:56.951824 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:56.951835 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:56.951856 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:56.952664 (+ 808us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:56.952775 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:56.952809 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:56.952942 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:56.953154 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 17:39:56.953484 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":330,"threads_started":1}
I20260923 17:39:56.956178 9937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2: Bootstrap starting.
I20260923 17:39:56.958659 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:39:56.959487 9937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:56.960520 9937 log.cc:824] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:56.963850 9937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2: No bootstrap required, opened a new log
I20260923 17:39:56.970880 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:56.964729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49889 (local address 127.0.0.1:34707)
0923 17:39:56.964948 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:56.964953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:56.965398 (+ 445us) server_negotiation.cc:407] Connection header received
0923 17:39:56.965606 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:56.965609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:56.965683 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:56.965774 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:56.966272 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:56.967022 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:56.968350 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:39:56.968841 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:39:56.968961 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:39:56.970136 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:56.970140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:56.970144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:56.970165 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:56.970213 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:56.970229 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:56.970319 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:56.970457 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 17:39:56.970564 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260923 17:39:56.972376 9937 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd12601081d74313993886449a9996e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } }
I20260923 17:39:56.972803 9937 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:56.972908 9937 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd12601081d74313993886449a9996e2, State: Initialized, Role: FOLLOWER
I20260923 17:39:56.973491 9937 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd12601081d74313993886449a9996e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } }
I20260923 17:39:56.973708 9937 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:39:56.973855 9937 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:39:56.974011 9937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:56.976390 9937 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd12601081d74313993886449a9996e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } }
I20260923 17:39:56.976935 9937 leader_election.cc:302] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd12601081d74313993886449a9996e2; no voters:
I20260923 17:39:56.977501 9937 leader_election.cc:288] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:39:56.977632 9947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:56.978284 9947 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [term 1 LEADER]: Becoming Leader. State: Replica: fd12601081d74313993886449a9996e2, State: Running, Role: LEADER
I20260923 17:39:56.978825 9947 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd12601081d74313993886449a9996e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } }
I20260923 17:39:56.979620 9937 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:39:56.981557 9949 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd12601081d74313993886449a9996e2. Latest consensus state: current_term: 1 leader_uuid: "fd12601081d74313993886449a9996e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd12601081d74313993886449a9996e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } } }
I20260923 17:39:56.981590 9948 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd12601081d74313993886449a9996e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd12601081d74313993886449a9996e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34707 } } }
I20260923 17:39:56.982223 9948 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:56.982254 9949 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2 [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:56.988694 9954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:39:56.994156 9954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:39:57.003312 9954 catalog_manager.cc:1347] Generated new cluster ID: c8de3ffeb8f04e7382a52dfd1dde27ef
I20260923 17:39:57.003443 9954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:39:57.024644 9954 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:39:57.026094 9954 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:39:57.055855 9954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd12601081d74313993886449a9996e2: Generated new TSK 0
I20260923 17:39:57.056788 9954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:39:57.071455 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 9870
W20260923 17:39:57.072881 9932 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46727
W20260923 17:39:57.073716 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 9870: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:39:57.073899 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 9870
2026-09-23T17:39:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1849 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:39:57 dist-test-slave-ppjl krb5kdc[9968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:39:57 dist-test-slave-ppjl krb5kdc[9968](info): set up 2 sockets
Sep 23 17:39:57 dist-test-slave-ppjl krb5kdc[9968](info): commencing operation
krb5kdc: starting...
W20260923 17:39:58.108503 21441 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:39:58 dist-test-slave-ppjl krb5kdc[9968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:39:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:39:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:39:58.365026 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43227
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:39:58.697606 9981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:39:58.697942 9981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:39:58.698064 9981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:39:58.705607 9981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:39:58.705749 9981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:39:58.705803 9981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:39:58.705888 9981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:39:58.705945 9981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:39:58.716426 9981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:39:58.717805 9981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:39:58.719223 9981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:39:58.731832 9986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:58.731940 9989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:39:58.731931 9987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:39:58.732833 9981 server_base.cc:1034] running on GCE node
I20260923 17:39:58.733705 9981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:39:58.735251 9981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:39:58.736481 9981 hybrid_clock.cc:648] HybridClock initialized: now 1790185198736424 us; error 50 us; skew 500 ppm
I20260923 17:39:58.736858 9981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:39:58 dist-test-slave-ppjl krb5kdc[9968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:39:58.744002 9981 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:39:58.745636 9981 webserver.cc:460] Webserver started at http://127.0.0.1:33605/ using document root <none> and password file <none>
I20260923 17:39:58.746366 9981 fs_manager.cc:359] Metadata directory not provided
I20260923 17:39:58.746483 9981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:39:58.746809 9981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:39:58.750221 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "0cd511d3288e44bba0871d2367ddc602"
format_stamp: "Formatted at 2026-09-23 17:39:58 on dist-test-slave-ppjl"
server_key: "a0242c6895dc99e2be49d4ed283974c7"
server_key_iv: "4730f32594f9fc4cb270f9504c39772d"
server_key_version: "encryptionkey@0"
I20260923 17:39:58.751034 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "0cd511d3288e44bba0871d2367ddc602"
format_stamp: "Formatted at 2026-09-23 17:39:58 on dist-test-slave-ppjl"
server_key: "a0242c6895dc99e2be49d4ed283974c7"
server_key_iv: "4730f32594f9fc4cb270f9504c39772d"
server_key_version: "encryptionkey@0"
I20260923 17:39:58.756878 9981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:39:58.760977 9996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:39:58.762424 9981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:39:58.762583 9981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "0cd511d3288e44bba0871d2367ddc602"
format_stamp: "Formatted at 2026-09-23 17:39:58 on dist-test-slave-ppjl"
server_key: "a0242c6895dc99e2be49d4ed283974c7"
server_key_iv: "4730f32594f9fc4cb270f9504c39772d"
server_key_version: "encryptionkey@0"
I20260923 17:39:58.762768 9981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:39:58.768790 9981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:39:58.773514 9981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:39:58.773790 9981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:39:58.792006 9981 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43227
I20260923 17:39:58.792083 10048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43227 every 8 connection(s)
I20260923 17:39:58.793411 9981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:39:58.797173 10049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:39:58.800985 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 9981
I20260923 17:39:58.801182 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:39:58.801733 21441 external_mini_cluster.cc:1428] Setting key 8a0e0642bff6b3c89463fec702135eed
I20260923 17:39:58.809989 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:58.803810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37791 (local address 127.0.0.1:43227)
0923 17:39:58.804610 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:58.804623 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:58.804738 (+ 115us) server_negotiation.cc:407] Connection header received
0923 17:39:58.806384 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:58.806408 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:58.806854 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:58.807284 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:58.808364 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:58.808371 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:58.808384 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:58.808399 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:58.808462 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:58.808483 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:58.808697 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:58.808854 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 17:39:58.809039 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":360,"threads_started":1}
I20260923 17:39:58.810079 10049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602: Bootstrap starting.
I20260923 17:39:58.813815 10049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602: Neither blocks nor log segments found. Creating new log.
I20260923 17:39:58.814283 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:39:58.814972 10049 log.cc:824] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602: Log is configured to *not* fsync() on all Append() calls
I20260923 17:39:58.818049 10049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602: No bootstrap required, opened a new log
I20260923 17:39:58.821662 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:39:58.818803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34069 (local address 127.0.0.1:43227)
0923 17:39:58.819008 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 17:39:58.819014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:39:58.819325 (+ 311us) server_negotiation.cc:407] Connection header received
0923 17:39:58.819553 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:39:58.819566 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:39:58.819627 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:39:58.819841 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:39:58.820902 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:39:58.820907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:39:58.820910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:39:58.820929 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:39:58.820975 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:39:58.820992 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:39:58.821098 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:39:58.821277 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 17:39:58.821344 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:39:58.826617 10049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cd511d3288e44bba0871d2367ddc602" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43227 } }
I20260923 17:39:58.827028 10049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:39:58.827106 10049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cd511d3288e44bba0871d2367ddc602, State: Initialized, Role: FOLLOWER
I20260923 17:39:58.827741 10049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cd511d3288e44bba0871d2367ddc602" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43227 } }
I20260923 17:39:58.827986 10049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:39:58.828130 10049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:39:58.828342 10049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:39:58.830745 10049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cd511d3288e44bba0871d2367ddc602" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43227 } }
I20260923 17:39:58.831286 10049 leader_election.cc:302] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cd511d3288e44bba0871d2367ddc602; no voters:
I20260923 17:39:58.831883 10049 leader_election.cc:288] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:39:58.832039 10059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:39:58.832698 10059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [term 1 LEADER]: Becoming Leader. State: Replica: 0cd511d3288e44bba0871d2367ddc602, State: Running, Role: LEADER
I20260923 17:39:58.833364 10059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cd511d3288e44bba0871d2367ddc602" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43227 } }
I20260923 17:39:58.834039 10049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:39:58.836843 10061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cd511d3288e44bba0871d2367ddc602. Latest consensus state: current_term: 1 leader_uuid: "0cd511d3288e44bba0871d2367ddc602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cd511d3288e44bba0871d2367ddc602" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43227 } } }
I20260923 17:39:58.836601 10060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cd511d3288e44bba0871d2367ddc602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cd511d3288e44bba0871d2367ddc602" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43227 } } }
I20260923 17:39:58.837203 10060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:58.837815 10061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602 [sys.catalog]: This master's current role is: LEADER
I20260923 17:39:58.838935 10066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:39:58.844666 10066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:39:58.853735 10066 catalog_manager.cc:1347] Generated new cluster ID: deb5dccc9b4347a28986c398a6488d1c
I20260923 17:39:58.853868 10066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:39:58.870378 10066 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:39:58.870952 10066 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:39:58.878810 10066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cd511d3288e44bba0871d2367ddc602: Generated new TSK 0
I20260923 17:39:58.880055 10066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:39:58.895293 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 9981
W20260923 17:39:58.896494 10044 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37791
W20260923 17:39:58.897249 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 9981: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:39:58.897392 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 9981
2026-09-23T17:39:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1823 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:39:58 dist-test-slave-ppjl krb5kdc[10080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:39:58 dist-test-slave-ppjl krb5kdc[10080](info): set up 2 sockets
Sep 23 17:39:58 dist-test-slave-ppjl krb5kdc[10080](info): commencing operation
krb5kdc: starting...
W20260923 17:39:59.940122 21441 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:40:00 dist-test-slave-ppjl krb5kdc[10080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:40:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:40:00.204105 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:40:00.538177 10093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:00.538537 10093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:00.538683 10093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:40:00.546268 10093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:00.546428 10093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:00.546514 10093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:00.546600 10093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:00.546661 10093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:00.556955 10093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:00.558368 10093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:00.559820 10093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:00.572227 10101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:00.572232 10098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:00.572319 10099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:00.573657 10093 server_base.cc:1034] running on GCE node
I20260923 17:40:00.574527 10093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:00.576146 10093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:00.577370 10093 hybrid_clock.cc:648] HybridClock initialized: now 1790185200577326 us; error 35 us; skew 500 ppm
I20260923 17:40:00.577765 10093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:00 dist-test-slave-ppjl krb5kdc[10080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:00.581053 10093 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:40:00.582906 10093 webserver.cc:460] Webserver started at http://127.0.0.1:39297/ using document root <none> and password file <none>
I20260923 17:40:00.583643 10093 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:00.583761 10093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:00.584061 10093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:00.587393 10093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "892ec65e400c42edaa0198361c0ca897"
format_stamp: "Formatted at 2026-09-23 17:40:00 on dist-test-slave-ppjl"
server_key: "7470ab2f63da341b7435b0913c9b6576"
server_key_iv: "0c608f0f5b8efd60714bbbf4ca8ac2b9"
server_key_version: "encryptionkey@0"
I20260923 17:40:00.588243 10093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "892ec65e400c42edaa0198361c0ca897"
format_stamp: "Formatted at 2026-09-23 17:40:00 on dist-test-slave-ppjl"
server_key: "7470ab2f63da341b7435b0913c9b6576"
server_key_iv: "0c608f0f5b8efd60714bbbf4ca8ac2b9"
server_key_version: "encryptionkey@0"
I20260923 17:40:00.593767 10093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:40:00.597579 10108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:00.599215 10093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:40:00.599419 10093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "892ec65e400c42edaa0198361c0ca897"
format_stamp: "Formatted at 2026-09-23 17:40:00 on dist-test-slave-ppjl"
server_key: "7470ab2f63da341b7435b0913c9b6576"
server_key_iv: "0c608f0f5b8efd60714bbbf4ca8ac2b9"
server_key_version: "encryptionkey@0"
I20260923 17:40:00.599728 10093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:00.615434 10093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:00.620048 10093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:00.620330 10093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:00.638375 10093 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38411
I20260923 17:40:00.638413 10159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38411 every 8 connection(s)
I20260923 17:40:00.639910 10093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:40:00.644201 10160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:00.649209 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 10093
I20260923 17:40:00.649379 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:40:00.649839 21441 external_mini_cluster.cc:1428] Setting key 5e5a810549f01e315e1f9abb16b14f5c
I20260923 17:40:00.656553 10160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897: Bootstrap starting.
I20260923 17:40:00.660270 10160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:00.661288 10160 log.cc:824] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:00.664655 10160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897: No bootstrap required, opened a new log
Sep 23 17:40:00 dist-test-slave-ppjl krb5kdc[10080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:40:00.675405 10160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892ec65e400c42edaa0198361c0ca897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260923 17:40:00.675945 10160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:40:00.676064 10160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892ec65e400c42edaa0198361c0ca897, State: Initialized, Role: FOLLOWER
I20260923 17:40:00.676740 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:00.651785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:46651 (local address 127.0.0.1:38411)
0923 17:40:00.652632 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:00.652652 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:00.652711 (+ 59us) server_negotiation.cc:407] Connection header received
0923 17:40:00.655221 (+ 2510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:00.655252 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:00.655786 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:00.656248 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:00.657832 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:00.659861 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:00.661242 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:00.662239 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:00.662373 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:00.666620 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:00.666632 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:00.666657 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:00.666717 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:00.671096 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:00.672209 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:00.672219 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:00.672229 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:00.672494 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:00.673102 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:00.673108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:00.673111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:00.674061 (+ 950us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:00.674421 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:00.674871 (+ 450us) server_negotiation.cc:299] Negotiation successful
0923 17:40:00.675287 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":323,"threads_started":1}
I20260923 17:40:00.676789 10160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892ec65e400c42edaa0198361c0ca897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260923 17:40:00.677126 10160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:40:00.677275 10160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:40:00.677444 10160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:00.680120 10160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892ec65e400c42edaa0198361c0ca897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260923 17:40:00.680747 10160 leader_election.cc:302] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 892ec65e400c42edaa0198361c0ca897; no voters:
I20260923 17:40:00.681161 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:40:00.681408 10160 leader_election.cc:288] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:40:00.681560 10165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:40:00.682134 10165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [term 1 LEADER]: Becoming Leader. State: Replica: 892ec65e400c42edaa0198361c0ca897, State: Running, Role: LEADER
I20260923 17:40:00.682818 10165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892ec65e400c42edaa0198361c0ca897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } }
I20260923 17:40:00.686412 10167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "892ec65e400c42edaa0198361c0ca897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892ec65e400c42edaa0198361c0ca897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } } }
I20260923 17:40:00.686898 10167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:00.691630 10165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 892ec65e400c42edaa0198361c0ca897. Latest consensus state: current_term: 1 leader_uuid: "892ec65e400c42edaa0198361c0ca897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892ec65e400c42edaa0198361c0ca897" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38411 } } }
I20260923 17:40:00.692584 10165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:00.695923 10160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:00.696130 10171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:40:00.702037 10171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:40:00.715318 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:00.695517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:39667 (local address 127.0.0.1:38411)
0923 17:40:00.695686 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:00.695693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:00.696167 (+ 474us) server_negotiation.cc:407] Connection header received
0923 17:40:00.696405 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:00.696410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:00.696498 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:00.696618 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:00.698580 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:00.699846 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:00.701744 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:00.702438 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:00.702551 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:00.705716 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:00.705725 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:00.705729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:00.705787 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:00.710251 (+ 4464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:00.711594 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:00.711601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:00.711605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:00.711804 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:00.712447 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:00.712454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:00.712458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:00.712986 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:00.713300 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:00.714606 (+ 1306us) server_negotiation.cc:299] Negotiation successful
0923 17:40:00.714828 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 17:40:00.728377 10171 catalog_manager.cc:1347] Generated new cluster ID: 4a2d2b45dbc2470ea666e9d146b77eee
I20260923 17:40:00.728559 10171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:40:00.782969 10171 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:40:00.784494 10171 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:40:00.799904 10171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 892ec65e400c42edaa0198361c0ca897: Generated new TSK 0
I20260923 17:40:00.800885 10171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:40:00.807626 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 10093
I20260923 17:40:00.808851 10155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:40:00.906335 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 10093
W20260923 17:40:00.921312 10078 connection.cc:535] client connection to 127.0.0.1:38411 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38411): Connection reset by peer (error 104)
2026-09-23T17:40:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2009 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:40:01 dist-test-slave-ppjl krb5kdc[10193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:40:01 dist-test-slave-ppjl krb5kdc[10193](info): set up 2 sockets
Sep 23 17:40:01 dist-test-slave-ppjl krb5kdc[10193](info): commencing operation
krb5kdc: starting...
W20260923 17:40:01.936348 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:40:02 dist-test-slave-ppjl krb5kdc[10193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:40:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:40:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:40:02.200282 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:40:02.532825 10206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:02.533134 10206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:02.533262 10206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:40:02.541043 10206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:02.541172 10206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:02.541231 10206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:02.541324 10206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:02.541440 10206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:02.551698 10206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:02.553040 10206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:02.554383 10206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:02.566778 10212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:02.566903 10214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:02.566839 10211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:02.567529 10206 server_base.cc:1034] running on GCE node
I20260923 17:40:02.568410 10206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:02.569950 10206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:02.571202 10206 hybrid_clock.cc:648] HybridClock initialized: now 1790185202571115 us; error 55 us; skew 500 ppm
I20260923 17:40:02.571570 10206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:02 dist-test-slave-ppjl krb5kdc[10193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:02.574999 10206 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:40:02.576574 10206 webserver.cc:460] Webserver started at http://127.0.0.1:36673/ using document root <none> and password file <none>
I20260923 17:40:02.577296 10206 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:02.577422 10206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:02.577764 10206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:02.581020 10206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "51d8560294b4470a9e88bda6f0a5e9e4"
format_stamp: "Formatted at 2026-09-23 17:40:02 on dist-test-slave-ppjl"
server_key: "4bce3a89bf7b832323029b890847480e"
server_key_iv: "f0a9060d7d0075f475bc743bc70c5c4c"
server_key_version: "encryptionkey@0"
I20260923 17:40:02.581802 10206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "51d8560294b4470a9e88bda6f0a5e9e4"
format_stamp: "Formatted at 2026-09-23 17:40:02 on dist-test-slave-ppjl"
server_key: "4bce3a89bf7b832323029b890847480e"
server_key_iv: "f0a9060d7d0075f475bc743bc70c5c4c"
server_key_version: "encryptionkey@0"
I20260923 17:40:02.587481 10206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:40:02.591421 10221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:02.592864 10206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:40:02.593030 10206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "51d8560294b4470a9e88bda6f0a5e9e4"
format_stamp: "Formatted at 2026-09-23 17:40:02 on dist-test-slave-ppjl"
server_key: "4bce3a89bf7b832323029b890847480e"
server_key_iv: "f0a9060d7d0075f475bc743bc70c5c4c"
server_key_version: "encryptionkey@0"
I20260923 17:40:02.593235 10206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:02.618669 10206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:02.623034 10206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:02.623342 10206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:02.641178 10206 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39509
I20260923 17:40:02.641196 10273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39509 every 8 connection(s)
I20260923 17:40:02.642669 10206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:40:02.645898 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 10206
I20260923 17:40:02.646121 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:40:02.646741 21441 external_mini_cluster.cc:1428] Setting key 61e410a39551a9090928b1a3226d6224
I20260923 17:40:02.647221 10274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:02.660799 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:02.648750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:51913 (local address 127.0.0.1:39509)
0923 17:40:02.649513 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:02.649525 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:02.649569 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:40:02.652118 (+ 2549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:02.652142 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:02.652590 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:02.652988 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:02.653550 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:02.654919 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:02.656284 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:02.656972 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:02.657141 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:02.658438 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:02.658446 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:02.658460 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:02.659033 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:02.659110 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:02.659134 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:02.659307 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:02.659583 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 17:40:02.659753 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":338,"threads_started":1}
I20260923 17:40:02.660923 10274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4: Bootstrap starting.
I20260923 17:40:02.664379 10274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:02.664669 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:40:02.665726 10274 log.cc:824] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:02.669291 10274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4: No bootstrap required, opened a new log
I20260923 17:40:02.676867 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:02.669690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:33463 (local address 127.0.0.1:39509)
0923 17:40:02.669897 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:02.669904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:02.670314 (+ 410us) server_negotiation.cc:407] Connection header received
0923 17:40:02.670526 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:02.670530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:02.670593 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:02.670669 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:02.671397 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:02.672182 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:02.674118 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:02.674581 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:02.674692 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:02.676058 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:02.676063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:02.676066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:02.676089 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:02.676148 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:02.676164 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:02.676253 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:02.676462 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 17:40:02.676509 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:40:02.679692 10274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d8560294b4470a9e88bda6f0a5e9e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260923 17:40:02.680135 10274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:40:02.680258 10274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d8560294b4470a9e88bda6f0a5e9e4, State: Initialized, Role: FOLLOWER
I20260923 17:40:02.680894 10274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d8560294b4470a9e88bda6f0a5e9e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260923 17:40:02.681154 10274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:40:02.681308 10274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:40:02.681483 10274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:02.684444 10274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d8560294b4470a9e88bda6f0a5e9e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260923 17:40:02.685313 10274 leader_election.cc:302] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51d8560294b4470a9e88bda6f0a5e9e4; no voters:
I20260923 17:40:02.686095 10274 leader_election.cc:288] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:40:02.686277 10284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:40:02.687431 10284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [term 1 LEADER]: Becoming Leader. State: Replica: 51d8560294b4470a9e88bda6f0a5e9e4, State: Running, Role: LEADER
I20260923 17:40:02.688014 10284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d8560294b4470a9e88bda6f0a5e9e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } }
I20260923 17:40:02.688493 10274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:02.691401 10286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51d8560294b4470a9e88bda6f0a5e9e4. Latest consensus state: current_term: 1 leader_uuid: "51d8560294b4470a9e88bda6f0a5e9e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d8560294b4470a9e88bda6f0a5e9e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } } }
I20260923 17:40:02.691839 10286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:02.692188 10285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51d8560294b4470a9e88bda6f0a5e9e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d8560294b4470a9e88bda6f0a5e9e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39509 } } }
I20260923 17:40:02.692497 10285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:02.693306 10293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:40:02.700030 10293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:40:02.710299 10293 catalog_manager.cc:1347] Generated new cluster ID: 89f47c3f86544111af25cd2428448b86
I20260923 17:40:02.710439 10293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:40:02.718550 10293 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:40:02.720542 10293 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:40:02.732470 10293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51d8560294b4470a9e88bda6f0a5e9e4: Generated new TSK 0
I20260923 17:40:02.733791 10293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:40:02.752336 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 10206
W20260923 17:40:02.753585 10269 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.183:51913
W20260923 17:40:02.754318 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 10206: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:40:02.754443 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 10206
W20260923 17:40:02.768652 10191 connection.cc:535] client connection to 127.0.0.1:39509 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39509): Connection reset by peer (error 104)
2026-09-23T17:40:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1847 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:40:02 dist-test-slave-ppjl krb5kdc[10306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:40:02 dist-test-slave-ppjl krb5kdc[10306](info): set up 2 sockets
Sep 23 17:40:02 dist-test-slave-ppjl krb5kdc[10306](info): commencing operation
krb5kdc: starting...
W20260923 17:40:03.776058 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:40:03 dist-test-slave-ppjl krb5kdc[10306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:40:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:40:04.035413 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44163
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:40:04.377959 10319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:04.378328 10319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:04.378479 10319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:40:04.385965 10319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:04.386121 10319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:04.386202 10319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:04.386289 10319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:04.386349 10319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:04.396845 10319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:04.398262 10319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:04.399622 10319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:04.411659 10324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:04.411624 10325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:04.411703 10327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:04.412220 10319 server_base.cc:1034] running on GCE node
I20260923 17:40:04.413074 10319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:04.414577 10319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:04.415843 10319 hybrid_clock.cc:648] HybridClock initialized: now 1790185204415788 us; error 41 us; skew 500 ppm
I20260923 17:40:04.416209 10319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:04 dist-test-slave-ppjl krb5kdc[10306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:04.427372 10319 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:40:04.429114 10319 webserver.cc:460] Webserver started at http://127.0.0.1:38623/ using document root <none> and password file <none>
I20260923 17:40:04.429894 10319 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:04.430030 10319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:04.430354 10319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:04.433622 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "8df60b4238564e1e994fcdc9850a592b"
format_stamp: "Formatted at 2026-09-23 17:40:04 on dist-test-slave-ppjl"
server_key: "972e17a275c3d308aea7ae8cdeb3d237"
server_key_iv: "90d1d337a2ee6e1ae482c270f20de9e3"
server_key_version: "encryptionkey@0"
I20260923 17:40:04.434464 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "8df60b4238564e1e994fcdc9850a592b"
format_stamp: "Formatted at 2026-09-23 17:40:04 on dist-test-slave-ppjl"
server_key: "972e17a275c3d308aea7ae8cdeb3d237"
server_key_iv: "90d1d337a2ee6e1ae482c270f20de9e3"
server_key_version: "encryptionkey@0"
I20260923 17:40:04.440486 10319 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260923 17:40:04.444406 10334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:04.445892 10319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:40:04.446084 10319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "8df60b4238564e1e994fcdc9850a592b"
format_stamp: "Formatted at 2026-09-23 17:40:04 on dist-test-slave-ppjl"
server_key: "972e17a275c3d308aea7ae8cdeb3d237"
server_key_iv: "90d1d337a2ee6e1ae482c270f20de9e3"
server_key_version: "encryptionkey@0"
I20260923 17:40:04.446372 10319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:04.452582 10319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:04.457238 10319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:04.457556 10319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:04.476354 10319 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44163
I20260923 17:40:04.476387 10386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44163 every 8 connection(s)
I20260923 17:40:04.477741 10319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:40:04.481118 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 10319
I20260923 17:40:04.481305 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:40:04.481832 21441 external_mini_cluster.cc:1428] Setting key bd043d885fe9f922848d84a6f499f81d
I20260923 17:40:04.482026 10387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:04.490432 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:04.483661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:46945 (local address 127.0.0.1:44163)
0923 17:40:04.484436 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:04.484447 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:04.484488 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:40:04.486629 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:04.486655 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:04.487047 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:04.487476 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:04.488668 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:04.488675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:04.488690 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:04.488710 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:04.488788 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:04.488812 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:04.489018 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:04.489127 (+ 109us) server_negotiation.cc:299] Negotiation successful
0923 17:40:04.489380 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":334,"threads_started":1}
I20260923 17:40:04.494277 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:40:04.495776 10387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b: Bootstrap starting.
I20260923 17:40:04.499061 10387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:04.500458 10387 log.cc:824] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:04.501087 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:04.498635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:53225 (local address 127.0.0.1:44163)
0923 17:40:04.498881 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:04.498887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:04.499194 (+ 307us) server_negotiation.cc:407] Connection header received
0923 17:40:04.499416 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:04.499420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:04.499477 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:04.499562 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:04.500464 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:04.500468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:04.500472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:04.500487 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:04.500527 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:04.500543 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:04.500651 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:04.500773 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 17:40:04.500825 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:40:04.505139 10387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b: No bootstrap required, opened a new log
I20260923 17:40:04.512822 10387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df60b4238564e1e994fcdc9850a592b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44163 } }
I20260923 17:40:04.513221 10387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:40:04.513332 10387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df60b4238564e1e994fcdc9850a592b, State: Initialized, Role: FOLLOWER
I20260923 17:40:04.513944 10387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df60b4238564e1e994fcdc9850a592b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44163 } }
I20260923 17:40:04.514159 10387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:40:04.514297 10387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:40:04.514446 10387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:04.516858 10387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df60b4238564e1e994fcdc9850a592b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44163 } }
I20260923 17:40:04.517395 10387 leader_election.cc:302] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8df60b4238564e1e994fcdc9850a592b; no voters:
I20260923 17:40:04.517972 10387 leader_election.cc:288] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:40:04.518117 10397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:40:04.518738 10397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [term 1 LEADER]: Becoming Leader. State: Replica: 8df60b4238564e1e994fcdc9850a592b, State: Running, Role: LEADER
I20260923 17:40:04.519372 10397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df60b4238564e1e994fcdc9850a592b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44163 } }
I20260923 17:40:04.520067 10387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:04.522526 10399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8df60b4238564e1e994fcdc9850a592b. Latest consensus state: current_term: 1 leader_uuid: "8df60b4238564e1e994fcdc9850a592b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df60b4238564e1e994fcdc9850a592b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44163 } } }
I20260923 17:40:04.522500 10398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8df60b4238564e1e994fcdc9850a592b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df60b4238564e1e994fcdc9850a592b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44163 } } }
I20260923 17:40:04.523020 10398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:04.523026 10399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:04.524001 10406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:40:04.529062 10406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:40:04.539149 10406 catalog_manager.cc:1347] Generated new cluster ID: dc765b07399a47fc82232cc3287f2000
I20260923 17:40:04.539315 10406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:40:04.562866 10406 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:40:04.563374 10406 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:40:04.573342 10406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8df60b4238564e1e994fcdc9850a592b: Generated new TSK 0
I20260923 17:40:04.574311 10406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:40:04.580096 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 10319
W20260923 17:40:04.581583 10382 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.183:46945
W20260923 17:40:04.582367 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 10319: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:40:04.582551 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 10319
2026-09-23T17:40:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1827 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:40:04 dist-test-slave-ppjl krb5kdc[10418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:40:04 dist-test-slave-ppjl krb5kdc[10418](info): set up 2 sockets
Sep 23 17:40:04 dist-test-slave-ppjl krb5kdc[10418](info): commencing operation
krb5kdc: starting...
W20260923 17:40:05.616199 21441 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:40:05 dist-test-slave-ppjl krb5kdc[10418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:40:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:40:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:40:05.878050 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:40:06.221886 10431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:06.222182 10431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:06.222302 10431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:40:06.229704 10431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:06.229838 10431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:06.229895 10431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:06.229985 10431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:06.230038 10431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:06.240118 10431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:06.241449 10431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:06.242771 10431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:06.254736 10437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:06.254907 10439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:06.254802 10436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:06.255501 10431 server_base.cc:1034] running on GCE node
I20260923 17:40:06.256359 10431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:06.257899 10431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:06.259258 10431 hybrid_clock.cc:648] HybridClock initialized: now 1790185206259200 us; error 45 us; skew 500 ppm
I20260923 17:40:06.259693 10431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:06 dist-test-slave-ppjl krb5kdc[10418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:06.263185 10431 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:40:06.264901 10431 webserver.cc:460] Webserver started at http://127.0.0.1:37011/ using document root <none> and password file <none>
I20260923 17:40:06.265641 10431 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:06.265769 10431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:06.266075 10431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:06.269332 10431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "46602c01fc264ae4b435a91f9195f9f3"
format_stamp: "Formatted at 2026-09-23 17:40:06 on dist-test-slave-ppjl"
server_key: "a728bff3428cf93017ab2b2edf38cead"
server_key_iv: "b6a5a2ed5b511e03b0c3269110259f35"
server_key_version: "encryptionkey@0"
I20260923 17:40:06.270156 10431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "46602c01fc264ae4b435a91f9195f9f3"
format_stamp: "Formatted at 2026-09-23 17:40:06 on dist-test-slave-ppjl"
server_key: "a728bff3428cf93017ab2b2edf38cead"
server_key_iv: "b6a5a2ed5b511e03b0c3269110259f35"
server_key_version: "encryptionkey@0"
I20260923 17:40:06.275868 10431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:40:06.279834 10447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:06.281337 10431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:40:06.281533 10431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "46602c01fc264ae4b435a91f9195f9f3"
format_stamp: "Formatted at 2026-09-23 17:40:06 on dist-test-slave-ppjl"
server_key: "a728bff3428cf93017ab2b2edf38cead"
server_key_iv: "b6a5a2ed5b511e03b0c3269110259f35"
server_key_version: "encryptionkey@0"
I20260923 17:40:06.281744 10431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:06.305568 10431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:06.310238 10431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:06.310528 10431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:06.328202 10431 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38041
I20260923 17:40:06.328234 10498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38041 every 8 connection(s)
I20260923 17:40:06.329629 10431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:40:06.333601 10499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:06.333886 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 10431
I20260923 17:40:06.334044 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:40:06.334467 21441 external_mini_cluster.cc:1428] Setting key 8d0295d968a6d31a3d810104f512e487
I20260923 17:40:06.348397 10499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3: Bootstrap starting.
Sep 23 17:40:06 dist-test-slave-ppjl krb5kdc[10418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:40:06.352319 10499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:06.353467 10499 log.cc:824] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:06.356339 10499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3: No bootstrap required, opened a new log
I20260923 17:40:06.358227 10502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:06.336372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:46823 (local address 127.0.0.1:38041)
0923 17:40:06.337225 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:06.337240 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:06.337299 (+ 59us) server_negotiation.cc:407] Connection header received
0923 17:40:06.338950 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:06.338980 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:06.339587 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:06.340067 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:06.341394 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:06.343353 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:06.344771 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:06.345884 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:06.346082 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:06.350696 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:06.350706 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:06.350729 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:06.350780 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:06.353933 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:06.354787 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:06.354794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:06.354801 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:06.354972 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:06.355499 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:06.355503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:06.355505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:06.356021 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:06.356337 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:06.356865 (+ 528us) server_negotiation.cc:299] Negotiation successful
0923 17:40:06.357195 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":342,"threads_started":1}
I20260923 17:40:06.362588 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:40:06.365156 10499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46602c01fc264ae4b435a91f9195f9f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38041 } }
I20260923 17:40:06.365628 10499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:40:06.365746 10499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46602c01fc264ae4b435a91f9195f9f3, State: Initialized, Role: FOLLOWER
I20260923 17:40:06.366333 10499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46602c01fc264ae4b435a91f9195f9f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38041 } }
I20260923 17:40:06.366568 10499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:40:06.366691 10499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:40:06.366860 10499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:06.370025 10499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46602c01fc264ae4b435a91f9195f9f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38041 } }
I20260923 17:40:06.370666 10499 leader_election.cc:302] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46602c01fc264ae4b435a91f9195f9f3; no voters:
I20260923 17:40:06.371289 10499 leader_election.cc:288] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:40:06.371423 10509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:40:06.372609 10509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [term 1 LEADER]: Becoming Leader. State: Replica: 46602c01fc264ae4b435a91f9195f9f3, State: Running, Role: LEADER
I20260923 17:40:06.373240 10509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46602c01fc264ae4b435a91f9195f9f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38041 } }
I20260923 17:40:06.373663 10499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:06.376358 10510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46602c01fc264ae4b435a91f9195f9f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46602c01fc264ae4b435a91f9195f9f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38041 } } }
I20260923 17:40:06.376821 10510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:06.376458 10511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46602c01fc264ae4b435a91f9195f9f3. Latest consensus state: current_term: 1 leader_uuid: "46602c01fc264ae4b435a91f9195f9f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46602c01fc264ae4b435a91f9195f9f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38041 } } }
I20260923 17:40:06.377480 10511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:06.383195 10515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:40:06.387217 10502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:06.367006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:43045 (local address 127.0.0.1:38041)
0923 17:40:06.367217 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:06.367223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:06.367948 (+ 725us) server_negotiation.cc:407] Connection header received
0923 17:40:06.368168 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:06.368172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:06.368249 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:06.368350 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:06.370323 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:06.371345 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:06.373093 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:06.373778 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:06.373912 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:06.379515 (+ 5603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:06.379522 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:06.379528 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:06.379585 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:06.383777 (+ 4192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:06.384792 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:06.384798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:06.384802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:06.384981 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:06.385584 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:06.385591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:06.385594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:06.385988 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:06.386332 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:06.386656 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 17:40:06.386843 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 17:40:06.388967 10515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:40:06.399091 10515 catalog_manager.cc:1347] Generated new cluster ID: ecf9a5e4d8e4440c8be3062116e2f735
I20260923 17:40:06.399248 10515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:40:06.421972 10515 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:40:06.423794 10515 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:40:06.433305 10515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46602c01fc264ae4b435a91f9195f9f3: Generated new TSK 0
I20260923 17:40:06.434186 10515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:40:06.446184 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 10431
I20260923 17:40:06.447440 10494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:40:06.541205 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 10431
W20260923 17:40:06.555884 10416 connection.cc:535] client connection to 127.0.0.1:38041 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38041): Connection reset by peer (error 104)
2026-09-23T17:40:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1958 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:40:06 dist-test-slave-ppjl krb5kdc[10531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:40:06 dist-test-slave-ppjl krb5kdc[10531](info): set up 2 sockets
Sep 23 17:40:06 dist-test-slave-ppjl krb5kdc[10531](info): commencing operation
krb5kdc: starting...
W20260923 17:40:07.568451 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:40:07 dist-test-slave-ppjl krb5kdc[10531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:40:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:40:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:40:07.827546 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35787
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:40:08.161677 10544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:08.161991 10544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:08.162127 10544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:40:08.169435 10544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:08.169571 10544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:08.169662 10544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:08.169745 10544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:08.169806 10544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:08.180071 10544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:08.181375 10544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:08.182727 10544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:08.194672 10550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:08.195091 10552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:08.195150 10549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:08.195911 10544 server_base.cc:1034] running on GCE node
I20260923 17:40:08.196779 10544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:08.198272 10544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:08.199482 10544 hybrid_clock.cc:648] HybridClock initialized: now 1790185208199423 us; error 52 us; skew 500 ppm
I20260923 17:40:08.199848 10544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:08 dist-test-slave-ppjl krb5kdc[10531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185208, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:08.202960 10544 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:40:08.204586 10544 webserver.cc:460] Webserver started at http://127.0.0.1:41269/ using document root <none> and password file <none>
I20260923 17:40:08.205291 10544 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:08.205410 10544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:08.205713 10544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:08.208841 10544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "2d2aab70bea74df484bc6f8fbeab7b64"
format_stamp: "Formatted at 2026-09-23 17:40:08 on dist-test-slave-ppjl"
server_key: "2942f16857622c43d52a258cdc0cf55c"
server_key_iv: "6b7613dfefdfff1562e956f3600a8ae0"
server_key_version: "encryptionkey@0"
I20260923 17:40:08.209609 10544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "2d2aab70bea74df484bc6f8fbeab7b64"
format_stamp: "Formatted at 2026-09-23 17:40:08 on dist-test-slave-ppjl"
server_key: "2942f16857622c43d52a258cdc0cf55c"
server_key_iv: "6b7613dfefdfff1562e956f3600a8ae0"
server_key_version: "encryptionkey@0"
I20260923 17:40:08.215078 10544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 17:40:08.218777 10559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:08.220324 10544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 17:40:08.220505 10544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "2d2aab70bea74df484bc6f8fbeab7b64"
format_stamp: "Formatted at 2026-09-23 17:40:08 on dist-test-slave-ppjl"
server_key: "2942f16857622c43d52a258cdc0cf55c"
server_key_iv: "6b7613dfefdfff1562e956f3600a8ae0"
server_key_version: "encryptionkey@0"
I20260923 17:40:08.220705 10544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:08.246973 10544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:08.251541 10544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:08.251866 10544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:08.268882 10544 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35787
I20260923 17:40:08.268913 10611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35787 every 8 connection(s)
I20260923 17:40:08.270220 10544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:40:08.273919 10612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:08.276682 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 10544
I20260923 17:40:08.276862 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:40:08.277380 21441 external_mini_cluster.cc:1428] Setting key 0368db427d480669ff000fa6f626df76
I20260923 17:40:08.286019 10612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64: Bootstrap starting.
I20260923 17:40:08.289252 10612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:08.290331 10612 log.cc:824] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:08.290706 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:08.279361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:41941 (local address 127.0.0.1:35787)
0923 17:40:08.280151 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:08.280167 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:08.280211 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:40:08.281964 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:08.281987 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:08.282416 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:08.282800 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:08.283383 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:08.284754 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:08.286055 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:08.286778 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:08.286942 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:08.288295 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:08.288302 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:08.288317 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:08.288881 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:08.288963 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:08.288988 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:08.289129 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:08.289432 (+ 303us) server_negotiation.cc:299] Negotiation successful
0923 17:40:08.289636 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":332,"threads_started":1}
I20260923 17:40:08.293450 10612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64: No bootstrap required, opened a new log
I20260923 17:40:08.294950 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:40:08.301582 10612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2aab70bea74df484bc6f8fbeab7b64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35787 } }
I20260923 17:40:08.302093 10612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:40:08.302207 10612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2aab70bea74df484bc6f8fbeab7b64, State: Initialized, Role: FOLLOWER
I20260923 17:40:08.302871 10612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2aab70bea74df484bc6f8fbeab7b64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35787 } }
I20260923 17:40:08.303186 10612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:40:08.303347 10612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:40:08.303516 10612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:08.305740 10615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:08.299250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:53563 (local address 127.0.0.1:35787)
0923 17:40:08.299449 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:08.299454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:08.299749 (+ 295us) server_negotiation.cc:407] Connection header received
0923 17:40:08.299928 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:08.299932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:08.300003 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:08.300090 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:08.300694 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:08.301439 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:08.303271 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:08.303809 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:08.303905 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:08.305057 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:08.305062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:08.305065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:08.305086 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:08.305136 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:08.305152 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:08.305237 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:08.305395 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 17:40:08.305438 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:40:08.306437 10612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2aab70bea74df484bc6f8fbeab7b64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35787 } }
I20260923 17:40:08.307042 10612 leader_election.cc:302] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d2aab70bea74df484bc6f8fbeab7b64; no voters:
I20260923 17:40:08.307672 10612 leader_election.cc:288] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:40:08.307799 10622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:40:08.308395 10622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [term 1 LEADER]: Becoming Leader. State: Replica: 2d2aab70bea74df484bc6f8fbeab7b64, State: Running, Role: LEADER
I20260923 17:40:08.308887 10622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2aab70bea74df484bc6f8fbeab7b64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35787 } }
I20260923 17:40:08.309653 10612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:08.311607 10623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d2aab70bea74df484bc6f8fbeab7b64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2aab70bea74df484bc6f8fbeab7b64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35787 } } }
I20260923 17:40:08.312084 10623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:08.313666 10624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d2aab70bea74df484bc6f8fbeab7b64. Latest consensus state: current_term: 1 leader_uuid: "2d2aab70bea74df484bc6f8fbeab7b64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2aab70bea74df484bc6f8fbeab7b64" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35787 } } }
I20260923 17:40:08.313988 10624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:08.315971 10627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:40:08.321678 10627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:40:08.330597 10627 catalog_manager.cc:1347] Generated new cluster ID: 963b7bce2ce3407e80a8c99d51b1fee3
I20260923 17:40:08.330731 10627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:40:08.350397 10627 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:40:08.351847 10627 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:40:08.359858 10627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d2aab70bea74df484bc6f8fbeab7b64: Generated new TSK 0
I20260923 17:40:08.360746 10627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:40:08.374572 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 10544
W20260923 17:40:08.375959 10607 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.183:41941
W20260923 17:40:08.376771 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 10544: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:40:08.376912 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 10544
W20260923 17:40:08.390793 10529 connection.cc:535] client connection to 127.0.0.1:35787 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35787): Connection reset by peer (error 104)
2026-09-23T17:40:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1835 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:40:08 dist-test-slave-ppjl krb5kdc[10643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:40:08 dist-test-slave-ppjl krb5kdc[10643](info): set up 2 sockets
Sep 23 17:40:08 dist-test-slave-ppjl krb5kdc[10643](info): commencing operation
krb5kdc: starting...
W20260923 17:40:09.404333 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:40:09 dist-test-slave-ppjl krb5kdc[10643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:40:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:40:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:40:09.670764 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:40:10.006253 10656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:10.006583 10656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:10.006690 10656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:40:10.014273 10656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:10.014411 10656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:10.014462 10656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:10.014523 10656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:10.014565 10656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:10.025343 10656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46883
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:10.026646 10656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:10.028026 10656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:10.039821 10661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:10.039871 10662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:10.039881 10664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:10.040508 10656 server_base.cc:1034] running on GCE node
I20260923 17:40:10.041332 10656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:10.042852 10656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:10.044102 10656 hybrid_clock.cc:648] HybridClock initialized: now 1790185210044029 us; error 38 us; skew 500 ppm
I20260923 17:40:10.044531 10656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:10 dist-test-slave-ppjl krb5kdc[10643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:10.047868 10656 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:40:10.049495 10656 webserver.cc:460] Webserver started at http://127.0.0.1:38827/ using document root <none> and password file <none>
I20260923 17:40:10.050235 10656 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:10.050369 10656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:10.050676 10656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:10.054081 10656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "a32955d5c2b646eeadca8c9bb0e1ed0a"
format_stamp: "Formatted at 2026-09-23 17:40:10 on dist-test-slave-ppjl"
server_key: "16f73d64f3d2d4b974dca82dadecc8d1"
server_key_iv: "fa486f20200aead279e8e609bc52314d"
server_key_version: "encryptionkey@0"
I20260923 17:40:10.054975 10656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "a32955d5c2b646eeadca8c9bb0e1ed0a"
format_stamp: "Formatted at 2026-09-23 17:40:10 on dist-test-slave-ppjl"
server_key: "16f73d64f3d2d4b974dca82dadecc8d1"
server_key_iv: "fa486f20200aead279e8e609bc52314d"
server_key_version: "encryptionkey@0"
I20260923 17:40:10.060700 10656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:40:10.064590 10671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:10.066164 10656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:40:10.066375 10656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "a32955d5c2b646eeadca8c9bb0e1ed0a"
format_stamp: "Formatted at 2026-09-23 17:40:10 on dist-test-slave-ppjl"
server_key: "16f73d64f3d2d4b974dca82dadecc8d1"
server_key_iv: "fa486f20200aead279e8e609bc52314d"
server_key_version: "encryptionkey@0"
I20260923 17:40:10.066581 10656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:10.072616 10656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:10.077204 10656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:10.077503 10656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:10.095618 10656 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46883
I20260923 17:40:10.095655 10722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46883 every 8 connection(s)
I20260923 17:40:10.097095 10656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:40:10.100975 10723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:10.104980 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 10656
I20260923 17:40:10.105140 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:40:10.105571 21441 external_mini_cluster.cc:1428] Setting key 3cdd174ed9f8fe935ef6820787c6e2fb
I20260923 17:40:10.113437 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:10.107393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:42881 (local address 127.0.0.1:46883)
0923 17:40:10.108152 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:10.108165 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:10.108207 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:40:10.109883 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:10.109909 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:10.110336 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:10.110707 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:10.111814 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:10.111821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:10.111836 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:10.111852 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:10.111914 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:10.111934 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:10.112116 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:10.112215 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 17:40:10.112458 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":323,"threads_started":1}
I20260923 17:40:10.114145 10723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a: Bootstrap starting.
I20260923 17:40:10.117309 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:40:10.117596 10723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:10.118598 10723 log.cc:824] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:10.122171 10723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a: No bootstrap required, opened a new log
I20260923 17:40:10.128052 10726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:10.125389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.183:47145 (local address 127.0.0.1:46883)
0923 17:40:10.125608 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:10.125614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:10.125962 (+ 348us) server_negotiation.cc:407] Connection header received
0923 17:40:10.126133 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:10.126137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:10.126189 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:10.126288 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:10.127346 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:10.127355 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:10.127358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:10.127376 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:10.127420 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:10.127438 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:10.127649 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:10.127688 (+ 39us) server_negotiation.cc:299] Negotiation successful
0923 17:40:10.127741 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 17:40:10.131278 10723 raft_consensus.cc:348] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32955d5c2b646eeadca8c9bb0e1ed0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46883 } }
I20260923 17:40:10.131696 10723 raft_consensus.cc:374] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:40:10.131779 10723 raft_consensus.cc:729] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a32955d5c2b646eeadca8c9bb0e1ed0a, State: Initialized, Role: FOLLOWER
I20260923 17:40:10.132332 10723 consensus_queue.cc:260] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32955d5c2b646eeadca8c9bb0e1ed0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46883 } }
I20260923 17:40:10.132557 10723 raft_consensus.cc:388] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:40:10.132659 10723 raft_consensus.cc:482] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:40:10.132800 10723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:10.135329 10723 raft_consensus.cc:504] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32955d5c2b646eeadca8c9bb0e1ed0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46883 } }
I20260923 17:40:10.135867 10723 leader_election.cc:302] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a32955d5c2b646eeadca8c9bb0e1ed0a; no voters:
I20260923 17:40:10.136588 10723 leader_election.cc:288] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:40:10.136759 10734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:40:10.137470 10734 raft_consensus.cc:686] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [term 1 LEADER]: Becoming Leader. State: Replica: a32955d5c2b646eeadca8c9bb0e1ed0a, State: Running, Role: LEADER
I20260923 17:40:10.138186 10734 consensus_queue.cc:237] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32955d5c2b646eeadca8c9bb0e1ed0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46883 } }
I20260923 17:40:10.141381 10723 sys_catalog.cc:564] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:10.141635 10736 sys_catalog.cc:455] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a32955d5c2b646eeadca8c9bb0e1ed0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32955d5c2b646eeadca8c9bb0e1ed0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46883 } } }
I20260923 17:40:10.142040 10736 sys_catalog.cc:458] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:10.144344 10735 sys_catalog.cc:455] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a32955d5c2b646eeadca8c9bb0e1ed0a. Latest consensus state: current_term: 1 leader_uuid: "a32955d5c2b646eeadca8c9bb0e1ed0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a32955d5c2b646eeadca8c9bb0e1ed0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46883 } } }
I20260923 17:40:10.144757 10735 sys_catalog.cc:458] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:10.151754 10738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:40:10.156082 10738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:40:10.164810 10738 catalog_manager.cc:1347] Generated new cluster ID: 4e7f6090c8184a4493c96a6a59c40671
I20260923 17:40:10.164966 10738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:40:10.181739 10738 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:40:10.182302 10738 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:40:10.200677 10738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a32955d5c2b646eeadca8c9bb0e1ed0a: Generated new TSK 0
I20260923 17:40:10.201947 10738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:40:10.214836 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 10656
W20260923 17:40:10.215993 10718 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.183:42881
W20260923 17:40:10.216694 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 10656: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:40:10.216817 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 10656
2026-09-23T17:40:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1839 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:40:10 dist-test-slave-ppjl krb5kdc[10755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:40:10 dist-test-slave-ppjl krb5kdc[10755](info): set up 2 sockets
Sep 23 17:40:10 dist-test-slave-ppjl krb5kdc[10755](info): commencing operation
krb5kdc: starting...
W20260923 17:40:11.240216 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:40:11 dist-test-slave-ppjl krb5kdc[10755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:40:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:40:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:40:11.494582 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41951
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:33823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:41951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:40:11.827845 10768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:11.828150 10768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:11.828282 10768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:40:11.835845 10768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:11.835978 10768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:11.836033 10768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:11.836119 10768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:11.836176 10768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:11.846446 10768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:33823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:41951
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:11.847942 10768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:11.849354 10768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:11.861506 10774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:11.861457 10773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:11.861557 10776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:11.862449 10768 server_base.cc:1034] running on GCE node
I20260923 17:40:11.863401 10768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:11.864975 10768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:11.866197 10768 hybrid_clock.cc:648] HybridClock initialized: now 1790185211866155 us; error 35 us; skew 500 ppm
I20260923 17:40:11.866549 10768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:11 dist-test-slave-ppjl krb5kdc[10755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185211, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:11.869783 10768 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:40:11.871345 10768 webserver.cc:460] Webserver started at http://127.20.240.126:43275/ using document root <none> and password file <none>
I20260923 17:40:11.872057 10768 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:11.872189 10768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:11.872496 10768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:11.875751 10768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "50edc8d4aaa940978f9f1d16275ca12a"
format_stamp: "Formatted at 2026-09-23 17:40:11 on dist-test-slave-ppjl"
server_key: "4fbb2147485996d80a89d3feb1336153"
server_key_iv: "3c7976a98fa0ad00855d63c554ba7474"
server_key_version: "encryptionkey@0"
I20260923 17:40:11.876576 10768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "50edc8d4aaa940978f9f1d16275ca12a"
format_stamp: "Formatted at 2026-09-23 17:40:11 on dist-test-slave-ppjl"
server_key: "4fbb2147485996d80a89d3feb1336153"
server_key_iv: "3c7976a98fa0ad00855d63c554ba7474"
server_key_version: "encryptionkey@0"
I20260923 17:40:11.881997 10768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260923 17:40:11.885636 10783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:11.887015 10768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 17:40:11.887224 10768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "50edc8d4aaa940978f9f1d16275ca12a"
format_stamp: "Formatted at 2026-09-23 17:40:11 on dist-test-slave-ppjl"
server_key: "4fbb2147485996d80a89d3feb1336153"
server_key_iv: "3c7976a98fa0ad00855d63c554ba7474"
server_key_version: "encryptionkey@0"
I20260923 17:40:11.887418 10768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:11.925653 10768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:11.930603 10768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:11.930920 10768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:11.949668 10768 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:41951
I20260923 17:40:11.949677 10834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:41951 every 8 connection(s)
I20260923 17:40:11.951051 10768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:40:11.954941 10835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:11.959972 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 10768
I20260923 17:40:11.960146 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:40:11.960556 21441 external_mini_cluster.cc:1428] Setting key 65910b6d6273bcf220a3f9d49b194b79
I20260923 17:40:11.966409 10835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a: Bootstrap starting.
I20260923 17:40:11.970254 10835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:11.971571 10835 log.cc:824] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:11.974491 10835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a: No bootstrap required, opened a new log
Sep 23 17:40:11 dist-test-slave-ppjl krb5kdc[10755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:40:11.982391 10835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50edc8d4aaa940978f9f1d16275ca12a" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41951 } }
I20260923 17:40:11.982892 10835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:40:11.983004 10835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50edc8d4aaa940978f9f1d16275ca12a, State: Initialized, Role: FOLLOWER
I20260923 17:40:11.983768 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:11.962445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58747 (local address 127.20.240.126:41951)
0923 17:40:11.963268 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:11.963285 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:11.963343 (+ 58us) server_negotiation.cc:407] Connection header received
0923 17:40:11.965117 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:11.965142 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:11.965569 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:11.965913 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:11.967749 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:11.969141 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:11.970920 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:11.971708 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:11.971895 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:11.976133 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:11.976141 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:11.976161 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:11.976205 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:11.979345 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:11.980177 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:11.980187 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:11.980194 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:11.980365 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:11.980833 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:11.980841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:11.980843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:11.981417 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:11.981727 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:11.982202 (+ 475us) server_negotiation.cc:299] Negotiation successful
0923 17:40:11.982543 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":345,"threads_started":1}
I20260923 17:40:11.983673 10835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50edc8d4aaa940978f9f1d16275ca12a" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41951 } }
I20260923 17:40:11.983911 10835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:40:11.984081 10835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:40:11.984270 10835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:11.986829 10835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50edc8d4aaa940978f9f1d16275ca12a" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41951 } }
I20260923 17:40:11.987507 10835 leader_election.cc:302] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50edc8d4aaa940978f9f1d16275ca12a; no voters:
I20260923 17:40:11.988015 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:40:11.988214 10835 leader_election.cc:288] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:40:11.988310 10840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:40:11.989440 10840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [term 1 LEADER]: Becoming Leader. State: Replica: 50edc8d4aaa940978f9f1d16275ca12a, State: Running, Role: LEADER
I20260923 17:40:11.990094 10840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50edc8d4aaa940978f9f1d16275ca12a" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41951 } }
I20260923 17:40:11.990751 10835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:11.993603 10842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50edc8d4aaa940978f9f1d16275ca12a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50edc8d4aaa940978f9f1d16275ca12a" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41951 } } }
I20260923 17:40:11.994078 10842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:11.994704 10841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50edc8d4aaa940978f9f1d16275ca12a. Latest consensus state: current_term: 1 leader_uuid: "50edc8d4aaa940978f9f1d16275ca12a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50edc8d4aaa940978f9f1d16275ca12a" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41951 } } }
I20260923 17:40:11.995000 10841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:12.004027 10850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:40:12.010501 10850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:40:12.026324 10850 catalog_manager.cc:1347] Generated new cluster ID: 2311a6b126be42caa7df5c2814e9ce9c
I20260923 17:40:12.026440 10850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:40:12.030056 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:12.007497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47859 (local address 127.20.240.126:41951)
0923 17:40:12.007704 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:12.007712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:12.013079 (+ 5367us) server_negotiation.cc:407] Connection header received
0923 17:40:12.013282 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:12.013288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:12.013376 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:12.013475 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:12.015184 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:12.016461 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:12.018552 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:12.019306 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:12.019419 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:12.023041 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:12.023049 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:12.023054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:12.023102 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:12.027005 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:12.027902 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:12.027907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:12.027910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:12.028039 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:12.028500 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:12.028503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:12.028505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:12.028858 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:12.029125 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:12.029517 (+ 392us) server_negotiation.cc:299] Negotiation successful
0923 17:40:12.029697 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:40:12.038897 10850 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:40:12.040374 10850 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:40:12.052510 10850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50edc8d4aaa940978f9f1d16275ca12a: Generated new TSK 0
I20260923 17:40:12.053402 10850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:40:12.066625 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 10768
I20260923 17:40:12.068653 10830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:40:12.163704 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 10768
W20260923 17:40:12.178328 10753 connection.cc:535] client connection to 127.20.240.126:41951 recv error: Network error: failed to read from TLS socket (remote: 127.20.240.126:41951): Connection reset by peer (error 104)
2026-09-23T17:40:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:40:12 dist-test-slave-ppjl krb5kdc[10867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:40:12 dist-test-slave-ppjl krb5kdc[10867](info): set up 2 sockets
Sep 23 17:40:12 dist-test-slave-ppjl krb5kdc[10867](info): commencing operation
krb5kdc: starting...
W20260923 17:40:13.188165 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:40:13 dist-test-slave-ppjl krb5kdc[10867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:40:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:40:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:40:13.442653 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:42397
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:38059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:42397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:40:13.777727 10881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:13.778036 10881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:13.778128 10881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:40:13.785437 10881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:13.785570 10881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:13.785617 10881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:13.785678 10881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:13.785723 10881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:13.795806 10881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:38059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:42397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:42397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:13.797262 10881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:13.798614 10881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:13.810856 10886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:13.810797 10887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:13.810895 10889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:13.811942 10881 server_base.cc:1034] running on GCE node
I20260923 17:40:13.812783 10881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:13.814301 10881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:13.815544 10881 hybrid_clock.cc:648] HybridClock initialized: now 1790185213815508 us; error 101 us; skew 500 ppm
I20260923 17:40:13.815917 10881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:13 dist-test-slave-ppjl krb5kdc[10867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185213, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:13.819211 10881 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:40:13.820760 10881 webserver.cc:460] Webserver started at http://127.20.240.126:38201/ using document root <none> and password file <none>
I20260923 17:40:13.821477 10881 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:13.821587 10881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:13.821888 10881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:13.825031 10881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "53d4e88960db4707866852c18a581384"
format_stamp: "Formatted at 2026-09-23 17:40:13 on dist-test-slave-ppjl"
server_key: "49309ec721e74d5a37d78ab80946e02b"
server_key_iv: "4bc33d0b58275037b64fc508ea3ce0c0"
server_key_version: "encryptionkey@0"
I20260923 17:40:13.825809 10881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "53d4e88960db4707866852c18a581384"
format_stamp: "Formatted at 2026-09-23 17:40:13 on dist-test-slave-ppjl"
server_key: "49309ec721e74d5a37d78ab80946e02b"
server_key_iv: "4bc33d0b58275037b64fc508ea3ce0c0"
server_key_version: "encryptionkey@0"
I20260923 17:40:13.831427 10881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:40:13.835263 10896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:13.836854 10881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:40:13.837036 10881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "53d4e88960db4707866852c18a581384"
format_stamp: "Formatted at 2026-09-23 17:40:13 on dist-test-slave-ppjl"
server_key: "49309ec721e74d5a37d78ab80946e02b"
server_key_iv: "4bc33d0b58275037b64fc508ea3ce0c0"
server_key_version: "encryptionkey@0"
I20260923 17:40:13.837255 10881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:13.862550 10881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:13.867133 10881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:13.867458 10881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:13.886273 10881 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:42397
I20260923 17:40:13.886288 10947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:42397 every 8 connection(s)
I20260923 17:40:13.887738 10881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:40:13.888016 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 10881
I20260923 17:40:13.888209 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:40:13.888772 21441 external_mini_cluster.cc:1428] Setting key 631ab4ed0bcd67701dfda092236cca01
I20260923 17:40:13.892134 10948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:13.905228 10951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:13.890921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39397 (local address 127.20.240.126:42397)
0923 17:40:13.891818 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:13.891833 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:13.892199 (+ 366us) server_negotiation.cc:407] Connection header received
0923 17:40:13.894614 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:13.894643 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:13.895196 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:13.895679 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:13.896215 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:13.898219 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:13.899786 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:13.900814 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:13.900996 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:13.902178 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:13.902188 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:13.902209 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:13.902960 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:13.903071 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:13.903107 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:13.903299 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:13.903594 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 17:40:13.903801 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":391,"threads_started":1}
I20260923 17:40:13.906188 10948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384: Bootstrap starting.
I20260923 17:40:13.909440 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:40:13.909674 10948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:13.910722 10948 log.cc:824] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:13.914038 10948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384: No bootstrap required, opened a new log
I20260923 17:40:13.920132 10951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:13.914092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48609 (local address 127.20.240.126:42397)
0923 17:40:13.914262 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:13.914269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:13.914700 (+ 431us) server_negotiation.cc:407] Connection header received
0923 17:40:13.914873 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:13.914877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:13.914938 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:13.915023 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:13.915536 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:13.916385 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:13.917677 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:13.918138 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:13.918282 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:13.919439 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:13.919443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:13.919447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:13.919467 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:13.919506 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:13.919520 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:13.919621 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:13.919817 (+ 196us) server_negotiation.cc:299] Negotiation successful
0923 17:40:13.919870 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:40:13.922343 10948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d4e88960db4707866852c18a581384" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42397 } }
I20260923 17:40:13.922703 10948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:40:13.922780 10948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d4e88960db4707866852c18a581384, State: Initialized, Role: FOLLOWER
I20260923 17:40:13.923378 10948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d4e88960db4707866852c18a581384" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42397 } }
I20260923 17:40:13.923588 10948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:40:13.923684 10948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:40:13.923812 10948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:13.926254 10948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d4e88960db4707866852c18a581384" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42397 } }
I20260923 17:40:13.926731 10948 leader_election.cc:302] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53d4e88960db4707866852c18a581384; no voters:
I20260923 17:40:13.927323 10948 leader_election.cc:288] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:40:13.927449 10958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:40:13.928120 10958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [term 1 LEADER]: Becoming Leader. State: Replica: 53d4e88960db4707866852c18a581384, State: Running, Role: LEADER
I20260923 17:40:13.928669 10958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d4e88960db4707866852c18a581384" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42397 } }
I20260923 17:40:13.929323 10948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:13.931329 10960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53d4e88960db4707866852c18a581384. Latest consensus state: current_term: 1 leader_uuid: "53d4e88960db4707866852c18a581384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d4e88960db4707866852c18a581384" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42397 } } }
I20260923 17:40:13.931288 10959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53d4e88960db4707866852c18a581384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53d4e88960db4707866852c18a581384" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 42397 } } }
I20260923 17:40:13.931739 10960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:13.931739 10959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:13.932727 10963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:40:13.939054 10963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:40:13.950636 10963 catalog_manager.cc:1347] Generated new cluster ID: e7c450ee1d8f4f1ba009fe83afe2fda5
I20260923 17:40:13.950773 10963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:40:13.982908 10963 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:40:13.984318 10963 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:40:13.993216 10963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53d4e88960db4707866852c18a581384: Generated new TSK 0
I20260923 17:40:13.994062 10963 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:40:14.004729 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 10881
W20260923 17:40:14.005980 10943 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39397
W20260923 17:40:14.006711 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 10881: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:40:14.006865 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 10881
W20260923 17:40:14.020962 10865 connection.cc:535] client connection to 127.20.240.126:42397 recv error: Network error: failed to read from TLS socket (remote: 127.20.240.126:42397): Connection reset by peer (error 104)
2026-09-23T17:40:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1842 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:40:14 dist-test-slave-ppjl krb5kdc[10979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:40:14 dist-test-slave-ppjl krb5kdc[10979](info): set up 2 sockets
Sep 23 17:40:14 dist-test-slave-ppjl krb5kdc[10979](info): commencing operation
krb5kdc: starting...
W20260923 17:40:15.044312 21441 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:40:15 dist-test-slave-ppjl krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:40:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:40:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:40:15.299196 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:44321
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:44321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:40:15.625367 10992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:15.625679 10992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:15.625788 10992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:40:15.633126 10992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:15.633246 10992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:15.633301 10992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:15.633359 10992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:15.633401 10992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:15.643518 10992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:44321
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:44321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:15.644805 10992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:15.646067 10992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:15.657362 10998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:15.657389 10997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:15.657510 11000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:15.657970 10992 server_base.cc:1034] running on GCE node
I20260923 17:40:15.658699 10992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:15.660074 10992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:15.661278 10992 hybrid_clock.cc:648] HybridClock initialized: now 1790185215661240 us; error 31 us; skew 500 ppm
I20260923 17:40:15.661612 10992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:15 dist-test-slave-ppjl krb5kdc[10979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185215, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:15.669157 10992 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:40:15.670634 10992 webserver.cc:460] Webserver started at http://127.20.240.126:34573/ using document root <none> and password file <none>
I20260923 17:40:15.671387 10992 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:15.671509 10992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:15.671800 10992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:15.674847 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "0f8fe0c5b3484d21b06427039d16fad3"
format_stamp: "Formatted at 2026-09-23 17:40:15 on dist-test-slave-ppjl"
server_key: "a49b937e5e5a7da74ea12feda6be0705"
server_key_iv: "3abd76f044b8cc28393f49f590e04348"
server_key_version: "encryptionkey@0"
I20260923 17:40:15.675606 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "0f8fe0c5b3484d21b06427039d16fad3"
format_stamp: "Formatted at 2026-09-23 17:40:15 on dist-test-slave-ppjl"
server_key: "a49b937e5e5a7da74ea12feda6be0705"
server_key_iv: "3abd76f044b8cc28393f49f590e04348"
server_key_version: "encryptionkey@0"
I20260923 17:40:15.681072 10992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:40:15.684695 11007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:15.686074 10992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:40:15.686245 10992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "0f8fe0c5b3484d21b06427039d16fad3"
format_stamp: "Formatted at 2026-09-23 17:40:15 on dist-test-slave-ppjl"
server_key: "a49b937e5e5a7da74ea12feda6be0705"
server_key_iv: "3abd76f044b8cc28393f49f590e04348"
server_key_version: "encryptionkey@0"
I20260923 17:40:15.686435 10992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:15.692482 10992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:15.696453 10992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:15.696724 10992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:15.714035 10992 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:44321
I20260923 17:40:15.714066 11058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:44321 every 8 connection(s)
I20260923 17:40:15.715430 10992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:40:15.719326 11059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:15.722779 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 10992
I20260923 17:40:15.722949 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:40:15.723552 21441 external_mini_cluster.cc:1428] Setting key 8eb1b9547470578d648b05c78c942d2f
I20260923 17:40:15.731148 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:15.725448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55579 (local address 127.20.240.126:44321)
0923 17:40:15.726212 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:15.726224 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:15.726265 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:40:15.727930 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:15.727953 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:15.728312 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:15.728668 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:15.729660 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:15.729666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:15.729679 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:15.729694 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:15.729755 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:15.729779 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:15.729976 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:15.730100 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 17:40:15.730277 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":325,"threads_started":1}
I20260923 17:40:15.731598 11059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3: Bootstrap starting.
I20260923 17:40:15.734536 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:40:15.734719 11059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:15.735742 11059 log.cc:824] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:15.738759 11059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3: No bootstrap required, opened a new log
I20260923 17:40:15.740922 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:15.738510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34059 (local address 127.20.240.126:44321)
0923 17:40:15.738649 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:15.738653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:15.739182 (+ 529us) server_negotiation.cc:407] Connection header received
0923 17:40:15.739344 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:15.739353 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:15.739397 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:15.739475 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:15.740304 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:15.740314 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:15.740319 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:15.740339 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:15.740378 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:15.740393 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:15.740497 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:15.740610 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 17:40:15.740672 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 17:40:15.746800 11059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8fe0c5b3484d21b06427039d16fad3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44321 } }
I20260923 17:40:15.747176 11059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:40:15.747272 11059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8fe0c5b3484d21b06427039d16fad3, State: Initialized, Role: FOLLOWER
I20260923 17:40:15.747916 11059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8fe0c5b3484d21b06427039d16fad3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44321 } }
I20260923 17:40:15.748155 11059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:40:15.748296 11059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:40:15.748431 11059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:15.750551 11059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8fe0c5b3484d21b06427039d16fad3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44321 } }
I20260923 17:40:15.751026 11059 leader_election.cc:302] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f8fe0c5b3484d21b06427039d16fad3; no voters:
I20260923 17:40:15.751690 11059 leader_election.cc:288] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:40:15.751807 11069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:40:15.752647 11069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [term 1 LEADER]: Becoming Leader. State: Replica: 0f8fe0c5b3484d21b06427039d16fad3, State: Running, Role: LEADER
I20260923 17:40:15.753216 11069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8fe0c5b3484d21b06427039d16fad3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44321 } }
I20260923 17:40:15.754006 11059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:15.756162 11071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f8fe0c5b3484d21b06427039d16fad3. Latest consensus state: current_term: 1 leader_uuid: "0f8fe0c5b3484d21b06427039d16fad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8fe0c5b3484d21b06427039d16fad3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44321 } } }
I20260923 17:40:15.756093 11070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f8fe0c5b3484d21b06427039d16fad3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f8fe0c5b3484d21b06427039d16fad3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44321 } } }
I20260923 17:40:15.756516 11071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:15.756592 11070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:15.758054 11075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:40:15.762823 11075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:40:15.772260 11075 catalog_manager.cc:1347] Generated new cluster ID: 33f4c58ab9184ad28f98575bd3f36ba7
I20260923 17:40:15.772390 11075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:40:15.784881 11075 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:40:15.785388 11075 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:40:15.804211 11075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f8fe0c5b3484d21b06427039d16fad3: Generated new TSK 0
I20260923 17:40:15.805389 11075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:40:15.813956 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 10992
W20260923 17:40:15.815095 11054 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55579
W20260923 17:40:15.815927 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 10992: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:40:15.816067 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 10992
2026-09-23T17:40:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1808 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:40:15 dist-test-slave-ppjl krb5kdc[11090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:40:15 dist-test-slave-ppjl krb5kdc[11090](info): set up 2 sockets
Sep 23 17:40:15 dist-test-slave-ppjl krb5kdc[11090](info): commencing operation
krb5kdc: starting...
W20260923 17:40:16.844201 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:40:17 dist-test-slave-ppjl krb5kdc[11090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:40:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:40:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:40:17.106734 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:43323
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:36571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:43323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:40:17.441825 11103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:17.442137 11103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:17.442232 11103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:40:17.449661 11103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:17.449784 11103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:17.449841 11103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:17.449927 11103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:17.449980 11103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:17.460304 11103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:43323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:43323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:17.461676 11103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:17.463003 11103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:17.474946 11108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:17.474969 11112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:17.474915 11109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:17.480804 11103 server_base.cc:1034] running on GCE node
I20260923 17:40:17.481613 11103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:17.483204 11103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:17.484440 11103 hybrid_clock.cc:648] HybridClock initialized: now 1790185217484390 us; error 53 us; skew 500 ppm
I20260923 17:40:17.484807 11103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:17 dist-test-slave-ppjl krb5kdc[11090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185217, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:17.488031 11103 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:40:17.489660 11103 webserver.cc:460] Webserver started at http://127.20.240.126:45587/ using document root <none> and password file <none>
I20260923 17:40:17.490371 11103 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:17.490491 11103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:17.490790 11103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:17.493824 11103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "0e87013b22254d0696dfd12990014ca4"
format_stamp: "Formatted at 2026-09-23 17:40:17 on dist-test-slave-ppjl"
server_key: "58d889af586f990b332e6aa284aef234"
server_key_iv: "f1bee773db1f8b60bdcc6dc6f2936fe2"
server_key_version: "encryptionkey@0"
I20260923 17:40:17.494565 11103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "0e87013b22254d0696dfd12990014ca4"
format_stamp: "Formatted at 2026-09-23 17:40:17 on dist-test-slave-ppjl"
server_key: "58d889af586f990b332e6aa284aef234"
server_key_iv: "f1bee773db1f8b60bdcc6dc6f2936fe2"
server_key_version: "encryptionkey@0"
I20260923 17:40:17.499871 11103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:40:17.503615 11119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:17.505069 11103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:40:17.505250 11103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "0e87013b22254d0696dfd12990014ca4"
format_stamp: "Formatted at 2026-09-23 17:40:17 on dist-test-slave-ppjl"
server_key: "58d889af586f990b332e6aa284aef234"
server_key_iv: "f1bee773db1f8b60bdcc6dc6f2936fe2"
server_key_version: "encryptionkey@0"
I20260923 17:40:17.505453 11103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:17.522421 11103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:17.526719 11103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:17.527025 11103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:17.545639 11103 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:43323
I20260923 17:40:17.545712 11171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:43323 every 8 connection(s)
I20260923 17:40:17.547048 11103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:40:17.551075 11172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:17.552094 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 11103
I20260923 17:40:17.552281 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:40:17.552830 21441 external_mini_cluster.cc:1428] Setting key 72f2a3857245b32119044088ae84d81e
I20260923 17:40:17.566300 11172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4: Bootstrap starting.
Sep 23 17:40:17 dist-test-slave-ppjl krb5kdc[11090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:40:17.570677 11172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:17.571980 11172 log.cc:824] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:17.575812 11172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4: No bootstrap required, opened a new log
I20260923 17:40:17.578970 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:17.554742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47399 (local address 127.20.240.126:43323)
0923 17:40:17.555592 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:17.555608 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:17.555653 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:40:17.557585 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:17.557611 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:17.558094 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:17.558504 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:17.560444 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:17.561889 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:17.563960 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:17.564941 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:17.565147 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:17.570380 (+ 5233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:17.570389 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:17.570408 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:17.570470 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:17.574257 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:17.575325 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:17.575335 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:17.575343 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:17.575592 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:17.576033 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:17.576037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:17.576039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:17.576686 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:17.576976 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:17.577416 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 17:40:17.577828 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":396,"threads_started":1}
I20260923 17:40:17.583226 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:40:17.585544 11172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e87013b22254d0696dfd12990014ca4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43323 } }
I20260923 17:40:17.586050 11172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:40:17.586156 11172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e87013b22254d0696dfd12990014ca4, State: Initialized, Role: FOLLOWER
I20260923 17:40:17.586858 11172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e87013b22254d0696dfd12990014ca4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43323 } }
I20260923 17:40:17.587191 11172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:40:17.587338 11172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:40:17.587505 11172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:17.590720 11172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e87013b22254d0696dfd12990014ca4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43323 } }
I20260923 17:40:17.591398 11172 leader_election.cc:302] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e87013b22254d0696dfd12990014ca4; no voters:
I20260923 17:40:17.592005 11172 leader_election.cc:288] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:40:17.592173 11182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:40:17.593950 11172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:17.594650 11182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [term 1 LEADER]: Becoming Leader. State: Replica: 0e87013b22254d0696dfd12990014ca4, State: Running, Role: LEADER
I20260923 17:40:17.595359 11182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e87013b22254d0696dfd12990014ca4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43323 } }
I20260923 17:40:17.598189 11183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e87013b22254d0696dfd12990014ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e87013b22254d0696dfd12990014ca4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43323 } } }
I20260923 17:40:17.598559 11183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:17.599125 11184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e87013b22254d0696dfd12990014ca4. Latest consensus state: current_term: 1 leader_uuid: "0e87013b22254d0696dfd12990014ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e87013b22254d0696dfd12990014ca4" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43323 } } }
I20260923 17:40:17.599447 11184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:17.603978 11190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:40:17.606343 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:17.588551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42987 (local address 127.20.240.126:43323)
0923 17:40:17.588770 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:17.588778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:17.589162 (+ 384us) server_negotiation.cc:407] Connection header received
0923 17:40:17.589328 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:17.589332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:17.589414 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:17.589497 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:17.591379 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:17.592372 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:17.594427 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:17.595124 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:17.595303 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:17.598636 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:17.598643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:17.598649 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:17.598696 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:17.602679 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:17.603724 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:17.603730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:17.603733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:17.603891 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:17.604335 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:17.604341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:17.604344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:17.604840 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:17.605112 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:17.605676 (+ 564us) server_negotiation.cc:299] Negotiation successful
0923 17:40:17.605933 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:40:17.611467 11190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:40:17.620675 11190 catalog_manager.cc:1347] Generated new cluster ID: 8e0aaa777b8c45d6afe2c4ed00e4da4d
I20260923 17:40:17.620788 11190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:40:17.656664 11190 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:40:17.658187 11190 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:40:17.669539 11190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e87013b22254d0696dfd12990014ca4: Generated new TSK 0
I20260923 17:40:17.670401 11190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:40:17.676847 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 11103
I20260923 17:40:17.678028 11167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:40:17.772284 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 11103
2026-09-23T17:40:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1956 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:40:17 dist-test-slave-ppjl krb5kdc[11204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:40:17 dist-test-slave-ppjl krb5kdc[11204](info): set up 2 sockets
Sep 23 17:40:17 dist-test-slave-ppjl krb5kdc[11204](info): commencing operation
krb5kdc: starting...
W20260923 17:40:18.800146 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:40:18 dist-test-slave-ppjl krb5kdc[11204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:40:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:40:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:40:19.058869 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41199
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:41199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:40:19.390662 11218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:19.390976 11218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:19.391098 11218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:40:19.398535 11218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:19.398659 11218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:19.398715 11218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:19.398800 11218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:19.398859 11218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:19.409050 11218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:41199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:41199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:19.410384 11218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:19.411783 11218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:19.423403 11223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:19.423429 11224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:19.423629 11226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:19.424239 11218 server_base.cc:1034] running on GCE node
I20260923 17:40:19.425061 11218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:19.426586 11218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:19.427810 11218 hybrid_clock.cc:648] HybridClock initialized: now 1790185219427748 us; error 55 us; skew 500 ppm
I20260923 17:40:19.428159 11218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:19 dist-test-slave-ppjl krb5kdc[11204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185219, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:19.431269 11218 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:40:19.432845 11218 webserver.cc:460] Webserver started at http://127.20.240.126:45093/ using document root <none> and password file <none>
I20260923 17:40:19.433511 11218 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:19.433611 11218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:19.433859 11218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:19.436877 11218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "409360697f59465586aa59c06154b435"
format_stamp: "Formatted at 2026-09-23 17:40:19 on dist-test-slave-ppjl"
server_key: "d5acc472b3b2b542139d96a6fa437ac8"
server_key_iv: "d8e97c32fa7838b0f3f86e1093f16516"
server_key_version: "encryptionkey@0"
I20260923 17:40:19.437606 11218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "409360697f59465586aa59c06154b435"
format_stamp: "Formatted at 2026-09-23 17:40:19 on dist-test-slave-ppjl"
server_key: "d5acc472b3b2b542139d96a6fa437ac8"
server_key_iv: "d8e97c32fa7838b0f3f86e1093f16516"
server_key_version: "encryptionkey@0"
I20260923 17:40:19.443008 11218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:40:19.446744 11233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:19.448280 11218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:40:19.448523 11218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "409360697f59465586aa59c06154b435"
format_stamp: "Formatted at 2026-09-23 17:40:19 on dist-test-slave-ppjl"
server_key: "d5acc472b3b2b542139d96a6fa437ac8"
server_key_iv: "d8e97c32fa7838b0f3f86e1093f16516"
server_key_version: "encryptionkey@0"
I20260923 17:40:19.448724 11218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:19.484576 11218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:19.489033 11218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:19.489282 11218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:19.506903 11218 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:41199
I20260923 17:40:19.506903 11284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:41199 every 8 connection(s)
I20260923 17:40:19.508419 11218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:40:19.512496 11285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:19.514657 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 11218
I20260923 17:40:19.514829 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:40:19.515343 21441 external_mini_cluster.cc:1428] Setting key ff86ee5899989f6839b7bc8cd06950e2
I20260923 17:40:19.526523 11285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435: Bootstrap starting.
I20260923 17:40:19.530750 11285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:19.531806 11289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:19.517349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58853 (local address 127.20.240.126:41199)
0923 17:40:19.518200 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:19.518220 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:19.518276 (+ 56us) server_negotiation.cc:407] Connection header received
0923 17:40:19.520604 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:19.520637 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:19.521214 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:19.521721 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:19.522340 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:19.524522 (+ 2182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:19.525995 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:19.527104 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:19.527344 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:19.528652 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:19.528665 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:19.528687 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:19.529453 (+ 766us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:19.529571 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:19.529608 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:19.529725 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:19.529984 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 17:40:19.530292 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":340,"threads_started":1}
I20260923 17:40:19.532210 11285 log.cc:824] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:19.535658 11285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435: No bootstrap required, opened a new log
I20260923 17:40:19.536337 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:40:19.544248 11285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409360697f59465586aa59c06154b435" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41199 } }
I20260923 17:40:19.544684 11285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:40:19.544785 11285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 409360697f59465586aa59c06154b435, State: Initialized, Role: FOLLOWER
I20260923 17:40:19.545542 11285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409360697f59465586aa59c06154b435" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41199 } }
I20260923 17:40:19.545883 11285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:40:19.546020 11285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:40:19.546186 11285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:19.547683 11289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:19.540679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40345 (local address 127.20.240.126:41199)
0923 17:40:19.540912 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:19.540919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:19.541451 (+ 532us) server_negotiation.cc:407] Connection header received
0923 17:40:19.541615 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:19.541621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:19.541708 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:19.541834 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:19.542353 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:19.543484 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:19.544890 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:19.545538 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:19.545673 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:19.546852 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:19.546859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:19.546864 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:19.546890 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:19.546949 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:19.546971 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:19.547065 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:19.547235 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 17:40:19.547347 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:40:19.549394 11285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409360697f59465586aa59c06154b435" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41199 } }
I20260923 17:40:19.549935 11285 leader_election.cc:302] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 409360697f59465586aa59c06154b435; no voters:
I20260923 17:40:19.550549 11285 leader_election.cc:288] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:40:19.550724 11296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:40:19.551375 11296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [term 1 LEADER]: Becoming Leader. State: Replica: 409360697f59465586aa59c06154b435, State: Running, Role: LEADER
I20260923 17:40:19.551957 11296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409360697f59465586aa59c06154b435" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41199 } }
I20260923 17:40:19.552649 11285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:19.559278 11298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 409360697f59465586aa59c06154b435. Latest consensus state: current_term: 1 leader_uuid: "409360697f59465586aa59c06154b435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409360697f59465586aa59c06154b435" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41199 } } }
I20260923 17:40:19.559736 11298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:19.559252 11297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "409360697f59465586aa59c06154b435" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "409360697f59465586aa59c06154b435" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 41199 } } }
I20260923 17:40:19.559916 11297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435 [sys.catalog]: This master's current role is: LEADER
W20260923 17:40:19.569932 11310 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:40:19.570086 11310 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:40:19.570211 11311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:40:19.572104 11311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:40:19.582082 11311 catalog_manager.cc:1347] Generated new cluster ID: 5f642ecc18e146ee910d26bde12a92e8
I20260923 17:40:19.582206 11311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:40:19.600768 11311 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:40:19.602819 11311 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:40:19.611652 11311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 409360697f59465586aa59c06154b435: Generated new TSK 0
I20260923 17:40:19.612912 11311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:40:19.628643 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 11218
W20260923 17:40:19.630182 11280 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58853
W20260923 17:40:19.630991 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 11218: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:40:19.631187 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 11218
2026-09-23T17:40:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1859 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:40:19 dist-test-slave-ppjl krb5kdc[11317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:40:19 dist-test-slave-ppjl krb5kdc[11317](info): set up 2 sockets
Sep 23 17:40:19 dist-test-slave-ppjl krb5kdc[11317](info): commencing operation
krb5kdc: starting...
W20260923 17:40:20.668093 21441 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:40:20 dist-test-slave-ppjl krb5kdc[11317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:40:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:40:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:40:20.928500 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:37523
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:37523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:40:21.267851 11330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:21.268177 11330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:21.268301 11330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:40:21.275786 11330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:21.275934 11330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:21.276008 11330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:21.276090 11330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:21.276149 11330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:21.286675 11330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:37523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:37523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:21.288091 11330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:21.289436 11330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:21.301084 11335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:21.301155 11336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:21.301133 11338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:21.302059 11330 server_base.cc:1034] running on GCE node
I20260923 17:40:21.302850 11330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:21.304419 11330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:21.305632 11330 hybrid_clock.cc:648] HybridClock initialized: now 1790185221305589 us; error 39 us; skew 500 ppm
I20260923 17:40:21.305989 11330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:21 dist-test-slave-ppjl krb5kdc[11317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185221, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:21.314318 11330 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:40:21.316133 11330 webserver.cc:460] Webserver started at http://127.20.240.126:33529/ using document root <none> and password file <none>
I20260923 17:40:21.316875 11330 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:21.316998 11330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:21.317327 11330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:21.320853 11330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "7078810e3c244f11b729320bfc97c8fa"
format_stamp: "Formatted at 2026-09-23 17:40:21 on dist-test-slave-ppjl"
server_key: "7a85b7e3d53bdcc2bcf6c0eee7759f6a"
server_key_iv: "1c89c5c48fca51e589cbbd90e86ab493"
server_key_version: "encryptionkey@0"
I20260923 17:40:21.321754 11330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "7078810e3c244f11b729320bfc97c8fa"
format_stamp: "Formatted at 2026-09-23 17:40:21 on dist-test-slave-ppjl"
server_key: "7a85b7e3d53bdcc2bcf6c0eee7759f6a"
server_key_iv: "1c89c5c48fca51e589cbbd90e86ab493"
server_key_version: "encryptionkey@0"
I20260923 17:40:21.327514 11330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:40:21.331564 11345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:21.333168 11330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:40:21.333374 11330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "7078810e3c244f11b729320bfc97c8fa"
format_stamp: "Formatted at 2026-09-23 17:40:21 on dist-test-slave-ppjl"
server_key: "7a85b7e3d53bdcc2bcf6c0eee7759f6a"
server_key_iv: "1c89c5c48fca51e589cbbd90e86ab493"
server_key_version: "encryptionkey@0"
I20260923 17:40:21.333611 11330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:21.340330 11330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:21.345464 11330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:21.345790 11330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:21.364828 11330 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:37523
I20260923 17:40:21.364857 11396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:37523 every 8 connection(s)
I20260923 17:40:21.366286 11330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:40:21.370153 11397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:21.374588 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 11330
I20260923 17:40:21.374769 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:40:21.375336 21441 external_mini_cluster.cc:1428] Setting key 50af9dc9ff11f6e896dceac4cd5fb540
I20260923 17:40:21.381795 11397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa: Bootstrap starting.
I20260923 17:40:21.383603 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:21.377252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32889 (local address 127.20.240.126:37523)
0923 17:40:21.378040 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:21.378053 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:21.378094 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:40:21.380236 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:21.380261 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:21.380617 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:21.380966 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:21.382033 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:21.382039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:21.382053 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:21.382067 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:21.382132 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:21.382152 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:21.382329 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:21.382425 (+ 96us) server_negotiation.cc:299] Negotiation successful
0923 17:40:21.382602 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":349,"threads_started":1}
I20260923 17:40:21.385172 11397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:21.386305 11397 log.cc:824] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:21.387434 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:40:21.389104 11397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa: No bootstrap required, opened a new log
I20260923 17:40:21.394426 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:21.391766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56749 (local address 127.20.240.126:37523)
0923 17:40:21.391968 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:21.391974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:21.392337 (+ 363us) server_negotiation.cc:407] Connection header received
0923 17:40:21.392526 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:21.392530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:21.392583 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:21.392686 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:21.393747 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:21.393752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:21.393755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:40:21.393773 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:21.393819 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:40:21.393836 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:21.393953 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:21.394068 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 17:40:21.394138 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:40:21.397559 11397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7078810e3c244f11b729320bfc97c8fa" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 37523 } }
I20260923 17:40:21.397989 11397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:40:21.398088 11397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7078810e3c244f11b729320bfc97c8fa, State: Initialized, Role: FOLLOWER
I20260923 17:40:21.398707 11397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7078810e3c244f11b729320bfc97c8fa" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 37523 } }
I20260923 17:40:21.398952 11397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:40:21.399080 11397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:40:21.399296 11397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:21.401847 11397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7078810e3c244f11b729320bfc97c8fa" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 37523 } }
I20260923 17:40:21.402354 11397 leader_election.cc:302] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7078810e3c244f11b729320bfc97c8fa; no voters:
I20260923 17:40:21.402938 11397 leader_election.cc:288] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:40:21.403072 11407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:40:21.403708 11407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [term 1 LEADER]: Becoming Leader. State: Replica: 7078810e3c244f11b729320bfc97c8fa, State: Running, Role: LEADER
I20260923 17:40:21.404244 11407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7078810e3c244f11b729320bfc97c8fa" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 37523 } }
I20260923 17:40:21.405089 11397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:21.407052 11408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7078810e3c244f11b729320bfc97c8fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7078810e3c244f11b729320bfc97c8fa" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 37523 } } }
I20260923 17:40:21.407099 11409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7078810e3c244f11b729320bfc97c8fa. Latest consensus state: current_term: 1 leader_uuid: "7078810e3c244f11b729320bfc97c8fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7078810e3c244f11b729320bfc97c8fa" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 37523 } } }
I20260923 17:40:21.407480 11408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:21.407486 11409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:21.409170 11413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:40:21.414649 11413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:40:21.423868 11413 catalog_manager.cc:1347] Generated new cluster ID: 34617b8c40994c298efb6020eecae4a6
I20260923 17:40:21.424000 11413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:40:21.438076 11413 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:40:21.438549 11413 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:40:21.446712 11413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7078810e3c244f11b729320bfc97c8fa: Generated new TSK 0
I20260923 17:40:21.447830 11413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:40:21.461196 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 11330
W20260923 17:40:21.462455 11392 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32889
W20260923 17:40:21.463341 21441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 11330: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:40:21.463510 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 11330
2026-09-23T17:40:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1831 ms)
[----------] 18 tests from AuthTokenIssuingTest (33903 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:40:21 dist-test-slave-ppjl krb5kdc[11429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:40:21 dist-test-slave-ppjl krb5kdc[11429](info): set up 2 sockets
Sep 23 17:40:21 dist-test-slave-ppjl krb5kdc[11429](info): commencing operation
krb5kdc: starting...
W20260923 17:40:22.488276 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:40:22 dist-test-slave-ppjl krb5kdc[11429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:40:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:40:22.747228 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:45519
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:45519,127.20.240.125:44647,127.20.240.124:37957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:40:23.081364 11442 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:40:23.081789 11442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:23.081892 11442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:23.082020 11442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:40:23.089600 11442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:23.089715 11442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:23.089794 11442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:23.089859 11442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:23.089955 11442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:23.100304 11442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:45519,127.20.240.125:44647,127.20.240.124:37957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:45519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:23.101678 11442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:23.103056 11442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:23.115077 11448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:23.115018 11450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:23.115103 11447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:23.116245 11442 server_base.cc:1034] running on GCE node
I20260923 17:40:23.117074 11442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:23.118609 11442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:23.119844 11442 hybrid_clock.cc:648] HybridClock initialized: now 1790185223119784 us; error 41 us; skew 500 ppm
I20260923 17:40:23.120203 11442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:23 dist-test-slave-ppjl krb5kdc[11429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185223, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:23.123627 11442 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:40:23.125253 11442 webserver.cc:460] Webserver started at http://127.20.240.126:40425/ using document root <none> and password file <none>
I20260923 17:40:23.125980 11442 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:23.126096 11442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:23.126415 11442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:23.129580 11442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "948b3642de9b4df4a75074b88ea94d9b"
format_stamp: "Formatted at 2026-09-23 17:40:23 on dist-test-slave-ppjl"
server_key: "a38f9b11b46368b26567fddd949cabf5"
server_key_iv: "38170027c9e4e9b49b45c3f62dc89083"
server_key_version: "encryptionkey@0"
I20260923 17:40:23.130451 11442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "948b3642de9b4df4a75074b88ea94d9b"
format_stamp: "Formatted at 2026-09-23 17:40:23 on dist-test-slave-ppjl"
server_key: "a38f9b11b46368b26567fddd949cabf5"
server_key_iv: "38170027c9e4e9b49b45c3f62dc89083"
server_key_version: "encryptionkey@0"
I20260923 17:40:23.136196 11442 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:40:23.139963 11457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:23.141582 11442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:40:23.141773 11442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "948b3642de9b4df4a75074b88ea94d9b"
format_stamp: "Formatted at 2026-09-23 17:40:23 on dist-test-slave-ppjl"
server_key: "a38f9b11b46368b26567fddd949cabf5"
server_key_iv: "38170027c9e4e9b49b45c3f62dc89083"
server_key_version: "encryptionkey@0"
I20260923 17:40:23.142004 11442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:23.158502 11442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:23.162880 11442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:23.163141 11442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:23.182781 11442 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:45519
I20260923 17:40:23.182801 11508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:45519 every 8 connection(s)
I20260923 17:40:23.184305 11442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:40:23.188381 11509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:23.192395 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 11442
I20260923 17:40:23.192585 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:40:23.193210 21441 external_mini_cluster.cc:1428] Setting key 89a5b13b9e4942984f4dd7f7beb681df
I20260923 17:40:23.195384 11509 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:40:23 dist-test-slave-ppjl krb5kdc[11429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185223, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.240.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.125@KRBTEST.COM" created.
I20260923 17:40:23.219096 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.200447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34262 (local address 127.20.240.126:45519)
0923 17:40:23.202240 (+ 1793us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:23.202249 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:23.202298 (+ 49us) server_negotiation.cc:407] Connection header received
0923 17:40:23.202694 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:23.202699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:23.202940 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:23.203124 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:23.204444 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.205680 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.207077 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.207696 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.207878 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:23.211413 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:23.211421 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:23.211436 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:23.211469 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:23.214659 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:23.215538 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:23.215547 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:23.215555 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:23.215721 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:23.216131 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:23.216134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:23.216137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:23.216772 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:23.216931 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:23.217557 (+ 626us) server_negotiation.cc:299] Negotiation successful
0923 17:40:23.217955 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":1600,"thread_start_us":1437,"threads_started":1}
I20260923 17:40:23.219144 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.199091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:45519 (local address 127.0.0.1:34262)
0923 17:40:23.199954 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.200000 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:23.202325 (+ 2325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:23.203206 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:23.203213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:23.203906 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:23.204244 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:23.204262 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.205831 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.205835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:23.206939 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:23.206952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.207932 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.207939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:23.208134 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:23.208813 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:23.208839 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:23.211180 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:23.214830 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:23.214843 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:23.214866 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:23.215349 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:23.215849 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:23.215852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:23.215855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:23.216027 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:23.216932 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:23.216939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:23.217127 (+ 188us) client_negotiation.cc:769] Sending connection context
0923 17:40:23.217338 (+ 211us) client_negotiation.cc:240] Negotiation successful
0923 17:40:23.217826 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":332,"threads_started":1}
I20260923 17:40:23.222254 11509 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:40:23.223544 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.223061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:44647 (local address 127.0.0.1:40230)
0923 17:40:23.223266 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.223402 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 17:40:23.223856 11459 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
I20260923 17:40:23.225448 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.225054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:44647 (local address 127.0.0.1:40240)
0923 17:40:23.225209 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.225297 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 17:40:23.225654 11509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.125:44647: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.125.keytab.
Entry for principal kudu/127.20.240.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.125.keytab.
I20260923 17:40:23.250135 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.125:44647
--webserver_interface=127.20.240.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.125.keytab
--principal=kudu/127.20.240.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:45519,127.20.240.125:44647,127.20.240.124:37957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:40:23.275002 11509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } attempt: 1
I20260923 17:40:23.276227 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.275772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:44647 (local address 127.0.0.1:40244)
0923 17:40:23.275970 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.276087 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260923 17:40:23.277177 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.276822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:44647 (local address 127.0.0.1:40248)
0923 17:40:23.276962 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.277040 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 17:40:23.277348 11509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.125:44647: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
I20260923 17:40:23.345633 11509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } attempt: 2
I20260923 17:40:23.346843 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.346260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:44647 (local address 127.0.0.1:40256)
0923 17:40:23.346570 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.346689 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 17:40:23.347810 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.347466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:44647 (local address 127.0.0.1:40258)
0923 17:40:23.347603 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.347686 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 17:40:23.347977 11509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.125:44647: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
I20260923 17:40:23.439247 11509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } attempt: 3
I20260923 17:40:23.440570 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.440083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:44647 (local address 127.0.0.1:40266)
0923 17:40:23.440313 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.440412 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260923 17:40:23.441668 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.441298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:44647 (local address 127.0.0.1:40268)
0923 17:40:23.441453 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.441525 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:40:23.441864 11509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.125:44647: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
W20260923 17:40:23.582885 11516 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:40:23.583310 11516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:23.583410 11516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:23.583523 11516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 17:40:23.585104 11509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } attempt: 4
I20260923 17:40:23.586249 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.585720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:44647 (local address 127.0.0.1:40270)
0923 17:40:23.586012 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.586104 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260923 17:40:23.587262 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.586855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:44647 (local address 127.0.0.1:40284)
0923 17:40:23.587002 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.587089 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 17:40:23.587469 11509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.125:44647: Network error: Client connection negotiation failed: client connection to 127.20.240.125:44647: connect: Connection refused (error 111)
W20260923 17:40:23.591337 11516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:23.591449 11516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:23.591547 11516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:23.591614 11516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:23.591698 11516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:23.602058 11516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:45519,127.20.240.125:44647,127.20.240.124:37957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.125.keytab
--principal=kudu/127.20.240.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.125:44647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.240.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:23.603521 11516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:23.605019 11516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:23.617219 11522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:23.617290 11521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:23.617257 11524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:23.618054 11516 server_base.cc:1034] running on GCE node
I20260923 17:40:23.619050 11516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:23.621186 11516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:23.622519 11516 hybrid_clock.cc:648] HybridClock initialized: now 1790185223622447 us; error 44 us; skew 500 ppm
I20260923 17:40:23.622912 11516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:23 dist-test-slave-ppjl krb5kdc[11429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185223, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:23.626408 11516 init.cc:377] Logged in from keytab as kudu/127.20.240.125@KRBTEST.COM (short username kudu)
I20260923 17:40:23.628170 11516 webserver.cc:460] Webserver started at http://127.20.240.125:33331/ using document root <none> and password file <none>
I20260923 17:40:23.628928 11516 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:23.629051 11516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:23.629387 11516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:23.632651 11516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/data/instance:
uuid: "62e679a38a28496a87d9ac7455854869"
format_stamp: "Formatted at 2026-09-23 17:40:23 on dist-test-slave-ppjl"
server_key: "f44ddde40a7ab02ff763edc617b868f2"
server_key_iv: "3f7c3f9d06844ddf728add0e890acc18"
server_key_version: "encryptionkey@0"
I20260923 17:40:23.633558 11516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/wal/instance:
uuid: "62e679a38a28496a87d9ac7455854869"
format_stamp: "Formatted at 2026-09-23 17:40:23 on dist-test-slave-ppjl"
server_key: "f44ddde40a7ab02ff763edc617b868f2"
server_key_iv: "3f7c3f9d06844ddf728add0e890acc18"
server_key_version: "encryptionkey@0"
I20260923 17:40:23.639343 11516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260923 17:40:23.643306 11531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:23.644776 11516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:40:23.644943 11516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/wal
uuid: "62e679a38a28496a87d9ac7455854869"
format_stamp: "Formatted at 2026-09-23 17:40:23 on dist-test-slave-ppjl"
server_key: "f44ddde40a7ab02ff763edc617b868f2"
server_key_iv: "3f7c3f9d06844ddf728add0e890acc18"
server_key_version: "encryptionkey@0"
I20260923 17:40:23.645165 11516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:23.666604 11516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:23.671096 11516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:23.671380 11516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:23.689496 11516 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.125:44647
I20260923 17:40:23.689584 11583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.125:44647 every 8 connection(s)
I20260923 17:40:23.690919 11516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/data/info.pb
I20260923 17:40:23.694730 11584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:23.696470 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 11516
I20260923 17:40:23.696663 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-1/wal/instance
I20260923 17:40:23.697227 21441 external_mini_cluster.cc:1428] Setting key de67f7ce20509a05dd49c7ec3d9242d8
I20260923 17:40:23.703428 11584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:40:23 dist-test-slave-ppjl krb5kdc[11429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185223, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.125@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.240.124@KRBTEST.COM; defaulting to no policy
I20260923 17:40:23.727345 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.706799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.20.240.126:45519)
0923 17:40:23.707025 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:23.707031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:23.708410 (+ 1379us) server_negotiation.cc:407] Connection header received
0923 17:40:23.711121 (+ 2711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:23.711126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:23.711241 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:23.711330 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:23.713310 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.714052 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.716164 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.716604 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.716704 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:23.720761 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:23.720768 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:23.720772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:23.720815 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:23.723714 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:23.724826 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:23.724831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:23.724833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:23.724980 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:23.725517 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:23.725520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:23.725522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:23.725850 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:23.725982 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:23.726827 (+ 845us) server_negotiation.cc:299] Negotiation successful
0923 17:40:23.727050 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.124@KRBTEST.COM" created.
I20260923 17:40:23.728494 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.707340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:45519 (local address 127.0.0.1:34278)
0923 17:40:23.708213 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.708263 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:23.710771 (+ 2508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:23.711617 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:23.711628 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:23.712371 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:23.713116 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:23.713141 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.714207 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.714212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:23.715982 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:23.715997 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.716819 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.716825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:23.717124 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:23.717834 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:23.717864 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:23.720485 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:23.723901 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:23.723912 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:23.723935 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:23.724663 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:23.725106 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:23.725111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:23.725115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:23.725384 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:23.725984 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:23.725992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:23.726376 (+ 384us) client_negotiation.cc:769] Sending connection context
0923 17:40:23.726698 (+ 322us) client_negotiation.cc:240] Negotiation successful
0923 17:40:23.727139 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":369,"threads_started":1}
I20260923 17:40:23.730547 11584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:40:23 dist-test-slave-ppjl krb5kdc[11429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185223, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.125@KRBTEST.COM for kudu/127.20.240.125@KRBTEST.COM
I20260923 17:40:23.746253 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.731437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:44647 (local address 127.0.0.1:40288)
0923 17:40:23.731638 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.731657 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:23.731770 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:23.732847 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:23.732851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:23.733237 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:23.733379 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:23.733387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.735248 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.735252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:23.736206 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:23.736220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.737271 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.737275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:23.737419 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:23.737996 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:23.738014 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:23.740371 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:23.743557 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:23.743563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:23.743567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:23.744036 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:23.744406 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:23.744410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:23.744413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:23.744581 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:23.745218 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:23.745222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:23.745362 (+ 140us) client_negotiation.cc:769] Sending connection context
0923 17:40:23.745485 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 17:40:23.745872 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 17:40:23.746338 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.731650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.20.240.125:44647)
0923 17:40:23.732330 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:23.732337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:23.732369 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:40:23.732421 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:23.732429 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:23.732663 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:23.732804 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:23.733550 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.735031 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.736338 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.737131 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.737247 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:23.740612 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:23.740626 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:23.740641 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:23.740677 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:23.743388 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:23.744186 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:23.744194 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:23.744196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:23.744317 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:23.744686 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:23.744689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:23.744691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:23.745097 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:23.745264 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:23.745630 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 17:40:23.746020 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":507,"thread_start_us":368,"threads_started":1}
I20260923 17:40:23.749215 11584 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:40:23.750710 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.750035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54814)
0923 17:40:23.750343 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.750535 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
W20260923 17:40:23.750989 11532 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
I20260923 17:40:23.752736 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.752323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54824)
0923 17:40:23.752519 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.752617 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260923 17:40:23.752971 11584 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:37957: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.124.keytab.
Entry for principal kudu/127.20.240.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.124.keytab.
I20260923 17:40:23.762928 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.124:37957
--webserver_interface=127.20.240.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.124.keytab
--principal=kudu/127.20.240.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:45519,127.20.240.125:44647,127.20.240.124:37957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:40:23.802299 11584 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } attempt: 1
I20260923 17:40:23.803689 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.803130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54830)
0923 17:40:23.803353 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.803524 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 17:40:23.804814 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.804374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54842)
0923 17:40:23.804572 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.804666 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260923 17:40:23.805055 11584 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:37957: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
I20260923 17:40:23.873296 11584 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } attempt: 2
I20260923 17:40:23.874432 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.873974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54852)
0923 17:40:23.874173 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.874266 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 17:40:23.875679 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.875285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54860)
0923 17:40:23.875442 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.875546 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 17:40:23.875926 11584 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:37957: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
I20260923 17:40:23.886734 11509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } attempt: 5
Sep 23 17:40:23 dist-test-slave-ppjl krb5kdc[11429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185223, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for kudu/127.20.240.125@KRBTEST.COM
I20260923 17:40:23.902462 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.887513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:44647 (local address 127.0.0.1:40302)
0923 17:40:23.887739 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.887762 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:23.887909 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:23.888291 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:23.888296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:23.888774 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:23.889033 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:23.889047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.890164 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.890171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:23.891662 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:23.891679 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.892591 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.892597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:23.892830 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:23.893468 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:23.893492 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:23.896019 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:23.899869 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:23.899873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:23.899877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:23.900287 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:23.900755 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:23.900760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:23.900762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:23.900925 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:23.901589 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:23.901594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:23.901751 (+ 157us) client_negotiation.cc:769] Sending connection context
0923 17:40:23.901882 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 17:40:23.902145 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 17:40:23.902602 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.887627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.20.240.125:44647)
0923 17:40:23.887840 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:23.887844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:23.887863 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:40:23.888069 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:23.888074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:23.888147 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:23.888236 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:23.889203 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.890010 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.891858 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:23.892414 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:23.892549 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:23.896272 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:23.896279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:23.896284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:23.896331 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:23.899674 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:23.900428 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:23.900434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:23.900438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:23.900606 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:23.901052 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:23.901058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:23.901061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:23.901436 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:23.901590 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:23.902022 (+ 432us) server_negotiation.cc:299] Negotiation successful
0923 17:40:23.902268 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 17:40:23.903824 11509 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:40:23.905102 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.904601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54868)
0923 17:40:23.904788 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.904924 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 17:40:23.906176 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.905786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54872)
0923 17:40:23.905936 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.906042 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 17:40:23.906406 11509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:37957: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
I20260923 17:40:23.957679 11509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } attempt: 1
I20260923 17:40:23.959462 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.958725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54882)
0923 17:40:23.959068 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.959236 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260923 17:40:23.960565 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.960171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54890)
0923 17:40:23.960335 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.960429 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:40:23.960780 11509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:37957: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
I20260923 17:40:23.967214 11584 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } attempt: 3
I20260923 17:40:23.968662 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.968097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54906)
0923 17:40:23.968382 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.968491 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260923 17:40:23.969933 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:23.969443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54916)
0923 17:40:23.969713 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:23.969802 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
W20260923 17:40:23.970193 11584 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:37957: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
I20260923 17:40:24.029045 11509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } attempt: 2
I20260923 17:40:24.030423 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.029820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54930)
0923 17:40:24.030093 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:24.030211 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 17:40:24.031745 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.031326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54936)
0923 17:40:24.031486 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:24.031586 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 17:40:24.031989 11509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:37957: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
W20260923 17:40:24.107973 11591 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:40:24.108306 11591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:24.108403 11591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:24.108529 11591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 17:40:24.113408 11584 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } attempt: 4
I20260923 17:40:24.114642 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.114126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54952)
0923 17:40:24.114373 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:24.114478 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 17:40:24.115675 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.115332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54954)
0923 17:40:24.115479 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:24.115559 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 17:40:24.115856 11584 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:37957: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
W20260923 17:40:24.116660 11591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:24.116739 11591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:24.116828 11591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:24.116890 11591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:24.116971 11591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:24.127343 11591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:45519,127.20.240.125:44647,127.20.240.124:37957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.124.keytab
--principal=kudu/127.20.240.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.124:37957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.240.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:24.128706 11591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:24.130048 11591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:40:24.138213 11509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } attempt: 3
I20260923 17:40:24.139513 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.138904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54960)
0923 17:40:24.139194 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:24.139310 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 17:40:24.140974 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.140463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54972)
0923 17:40:24.140732 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:24.140826 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260923 17:40:24.142622 11596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:24.142760 11509 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:37957: Network error: Client connection negotiation failed: client connection to 127.20.240.124:37957: connect: Connection refused (error 111)
W20260923 17:40:24.143842 11597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:24.144275 11599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:24.145126 11591 server_base.cc:1034] running on GCE node
I20260923 17:40:24.145912 11591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:24.147502 11591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:24.148710 11591 hybrid_clock.cc:648] HybridClock initialized: now 1790185224148644 us; error 41 us; skew 500 ppm
I20260923 17:40:24.149061 11591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:24 dist-test-slave-ppjl krb5kdc[11429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185224, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:24.152571 11591 init.cc:377] Logged in from keytab as kudu/127.20.240.124@KRBTEST.COM (short username kudu)
I20260923 17:40:24.154145 11591 webserver.cc:460] Webserver started at http://127.20.240.124:38319/ using document root <none> and password file <none>
I20260923 17:40:24.154917 11591 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:24.155014 11591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:24.155398 11591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:24.158599 11591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/data/instance:
uuid: "c3034379655644cab95402e0098fffa4"
format_stamp: "Formatted at 2026-09-23 17:40:24 on dist-test-slave-ppjl"
server_key: "fc2f74e917a5351cc02d0d4d7f9c0f8d"
server_key_iv: "32edf9b478bed93d901f52ea3138ea9f"
server_key_version: "encryptionkey@0"
I20260923 17:40:24.159546 11591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/wal/instance:
uuid: "c3034379655644cab95402e0098fffa4"
format_stamp: "Formatted at 2026-09-23 17:40:24 on dist-test-slave-ppjl"
server_key: "fc2f74e917a5351cc02d0d4d7f9c0f8d"
server_key_iv: "32edf9b478bed93d901f52ea3138ea9f"
server_key_version: "encryptionkey@0"
I20260923 17:40:24.165390 11591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:40:24.169344 11606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:24.171082 11591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:40:24.171324 11591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/wal
uuid: "c3034379655644cab95402e0098fffa4"
format_stamp: "Formatted at 2026-09-23 17:40:24 on dist-test-slave-ppjl"
server_key: "fc2f74e917a5351cc02d0d4d7f9c0f8d"
server_key_iv: "32edf9b478bed93d901f52ea3138ea9f"
server_key_version: "encryptionkey@0"
I20260923 17:40:24.171564 11591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:24.187886 11591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:24.192433 11591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:24.192714 11591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:24.210987 11591 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.124:37957
I20260923 17:40:24.211004 11657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.124:37957 every 8 connection(s)
I20260923 17:40:24.212419 11591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/data/info.pb
I20260923 17:40:24.216203 11658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:24.218704 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 11591
I20260923 17:40:24.218874 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184865977475-21441-0/minicluster-data/master-2/wal/instance
I20260923 17:40:24.219331 21441 external_mini_cluster.cc:1428] Setting key d6055ec33d8f1f36ea07276755b625a7
I20260923 17:40:24.223708 11658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:40:24 dist-test-slave-ppjl krb5kdc[11429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.124@KRBTEST.COM
Sep 23 17:40:24 dist-test-slave-ppjl krb5kdc[11429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185224, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.124@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:40:24.242734 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.226548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.20.240.126:45519)
0923 17:40:24.226760 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.226766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.227481 (+ 715us) server_negotiation.cc:407] Connection header received
0923 17:40:24.227732 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.227737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.227801 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.227885 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.229279 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.230298 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.232173 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.232770 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.232931 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.236227 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:24.236233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:24.236237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:24.236277 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:24.239329 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.240194 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.240199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.240201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.240363 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.240806 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.240810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.240812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.241154 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:24.241298 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:24.242140 (+ 842us) server_negotiation.cc:299] Negotiation successful
0923 17:40:24.242375 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:40:24.243760 11662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.226552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:45519 (local address 127.0.0.1:34292)
0923 17:40:24.227327 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:24.227380 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:24.227580 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:24.227980 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:24.227987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:24.228838 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:24.229095 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:24.229104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.230456 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.230460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:24.232035 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:24.232048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.232983 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.232989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:24.233109 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.233637 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:24.233659 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:24.236026 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:24.239472 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:24.239482 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:24.239493 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:24.240056 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:24.240498 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:24.240503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:24.240506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:24.240691 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:24.241276 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:24.241281 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:24.241725 (+ 444us) client_negotiation.cc:769] Sending connection context
0923 17:40:24.242035 (+ 310us) client_negotiation.cc:240] Negotiation successful
0923 17:40:24.242534 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":91,"spinlock_wait_cycles":18304,"thread_start_us":339,"threads_started":1}
I20260923 17:40:24.243794 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.221487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54978 (local address 127.20.240.124:37957)
0923 17:40:24.222284 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.222296 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.222340 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:40:24.224058 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.224082 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.224551 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.224948 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.226580 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.228583 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.230404 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.231528 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.231707 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.235870 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:24.235879 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:24.235894 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:24.235930 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:24.239491 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.240338 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.240347 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.240350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.240615 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.241015 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.241020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.241023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.241694 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:24.241931 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:24.242331 (+ 400us) server_negotiation.cc:299] Negotiation successful
0923 17:40:24.242658 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":576,"thread_start_us":353,"threads_started":1}
I20260923 17:40:24.245347 11658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:40:24 dist-test-slave-ppjl krb5kdc[11429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185224, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.124@KRBTEST.COM for kudu/127.20.240.125@KRBTEST.COM
Sep 23 17:40:24 dist-test-slave-ppjl krb5kdc[11429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:40:24.259522 11662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.245975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:44647 (local address 127.0.0.1:40316)
0923 17:40:24.246142 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:24.246178 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:24.246294 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:24.246639 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:24.246644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:24.247089 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:24.247300 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:24.247313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.248463 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.248469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:24.249552 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:24.249564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.250445 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.250452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:24.250655 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.251259 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:24.251282 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:24.253587 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:24.257056 (+ 3469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:24.257060 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:24.257062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:24.257520 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:24.257924 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:24.257927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:24.257929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:24.258089 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:24.258717 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:24.258720 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:24.258865 (+ 145us) client_negotiation.cc:769] Sending connection context
0923 17:40:24.259009 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 17:40:24.259270 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 17:40:24.259963 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.246056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.20.240.125:44647)
0923 17:40:24.246253 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.246259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.246278 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:40:24.246410 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.246415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.246498 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.246596 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.247459 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.248323 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.249688 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.250295 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.250384 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.253768 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:24.253773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:24.253776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:24.253808 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:24.256909 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.257642 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.257647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.257651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.257809 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.258207 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.258212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.258216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.258593 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:24.258741 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:24.259141 (+ 400us) server_negotiation.cc:299] Negotiation successful
0923 17:40:24.259401 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:40:24.260845 11658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:40:24.262620 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.247303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.20.240.126:45519)
0923 17:40:24.247490 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.247495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.247513 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:40:24.247585 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.247590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.247664 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.247752 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.248964 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.250046 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.251780 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.252213 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.252305 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.256426 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:24.256434 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:24.256439 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:24.256486 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:24.259751 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.260549 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.260554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.260557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.260790 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.261239 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.261244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.261247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.261570 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:24.261817 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:24.262118 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 17:40:24.262333 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 17:40:24 dist-test-slave-ppjl krb5kdc[11429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185224, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.124@KRBTEST.COM for kudu/127.20.240.124@KRBTEST.COM
Sep 23 17:40:24 dist-test-slave-ppjl krb5kdc[11429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.125@KRBTEST.COM
I20260923 17:40:24.274794 11662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.261628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54990)
0923 17:40:24.261840 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:24.261857 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:24.261986 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:24.262247 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:24.262251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:24.262701 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:24.262880 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:24.262889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.263950 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.263956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:24.264869 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:24.264881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.265534 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.265539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:24.265755 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.266267 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:24.266284 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:24.268351 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:24.272149 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:24.272155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:24.272160 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:24.272623 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:24.273071 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:24.273074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:24.273076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:24.273207 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:24.273806 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:24.273811 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:24.273948 (+ 137us) client_negotiation.cc:769] Sending connection context
0923 17:40:24.274065 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 17:40:24.274382 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":3}
I20260923 17:40:24.274818 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.261627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54990 (local address 127.20.240.124:37957)
0923 17:40:24.261799 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.261805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.261911 (+ 106us) server_negotiation.cc:407] Connection header received
0923 17:40:24.262079 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.262084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.262149 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.262221 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.263005 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.263845 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.265003 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.265393 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.265510 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.268654 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:24.268661 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:24.268667 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:24.268706 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:24.272000 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.272738 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.272744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.272747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.272943 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.273320 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.273325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.273332 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.273699 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:24.273830 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:24.274158 (+ 328us) server_negotiation.cc:299] Negotiation successful
0923 17:40:24.274535 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":158,"server-negotiator.queue_time_us":66}
I20260923 17:40:24.280062 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.264222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.20.240.125:44647)
0923 17:40:24.264379 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.264385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.264401 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:40:24.264451 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.264456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.264588 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.264668 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.265786 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.266499 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.267765 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.268380 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.268497 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.273792 (+ 5295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:24.273797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:24.273801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:24.273838 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:24.276932 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.277935 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.277940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.277942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.278115 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.278592 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.278596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.278599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.278891 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:24.279197 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:24.279561 (+ 364us) server_negotiation.cc:299] Negotiation successful
0923 17:40:24.279769 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 17:40:24.280833 11658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4: Bootstrap starting.
I20260923 17:40:24.284503 11658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:24.285218 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:40:24.285710 11658 log.cc:824] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:24.288695 11658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4: No bootstrap required, opened a new log
I20260923 17:40:24.288249 11484 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "948b3642de9b4df4a75074b88ea94d9b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34304
I20260923 17:40:24.296345 11658 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } }
I20260923 17:40:24.296700 11658 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3034379655644cab95402e0098fffa4, State: Initialized, Role: FOLLOWER
I20260923 17:40:24.297158 11658 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } }
I20260923 17:40:24.298131 11664 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } } }
I20260923 17:40:24.298514 11664 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:40:24.299104 11658 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:40:24.309126 11675 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:40:24.309276 11675 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:40:24.305944 11591 master_runner.cc:186] Error getting master registration for 127.20.240.126:45519: OK, instance_id { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" instance_seqno: 1790185223177635 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:40:24.316995 11591 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:40:24.319983 11509 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } attempt: 4
Sep 23 17:40:24 dist-test-slave-ppjl krb5kdc[11429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185223, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for kudu/127.20.240.124@KRBTEST.COM
I20260923 17:40:24.336872 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.320730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:54998)
0923 17:40:24.321029 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:24.321051 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:24.321175 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:24.321574 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:24.321581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:24.322093 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:24.322328 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:24.322342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.323858 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.323864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:24.325328 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:24.325347 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.326400 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.326406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:24.326628 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.327365 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:24.327391 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:24.330113 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:24.334175 (+ 4062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:24.334181 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:24.334186 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:24.334596 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:24.334996 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:24.335002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:24.335006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:24.335147 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:24.335864 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:24.335869 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:24.336046 (+ 177us) client_negotiation.cc:769] Sending connection context
0923 17:40:24.336171 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 17:40:24.336476 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260923 17:40:24.336872 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.320890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54998 (local address 127.20.240.124:37957)
0923 17:40:24.321080 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.321086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.321108 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:40:24.321322 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.321327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.321409 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.321506 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.322511 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.323678 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.325487 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.326241 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.326356 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.330311 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:24.330319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:24.330328 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:24.330371 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:24.334017 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.334720 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.334726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.334730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.334874 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.335267 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.335272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.335276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.335750 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:24.335918 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:24.336300 (+ 382us) server_negotiation.cc:299] Negotiation successful
0923 17:40:24.336512 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 17:40:24.342829 11509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b: Bootstrap starting.
I20260923 17:40:24.346979 11509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:24.348322 11509 log.cc:824] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:24.351615 11509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b: No bootstrap required, opened a new log
I20260923 17:40:24.359393 11509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } }
I20260923 17:40:24.359897 11509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 948b3642de9b4df4a75074b88ea94d9b, State: Initialized, Role: FOLLOWER
I20260923 17:40:24.360378 11509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } }
I20260923 17:40:24.361366 11678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } } }
I20260923 17:40:24.361794 11678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:40:24.362615 11509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:24.362762 11484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 17:40:24.362985 11484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:24.366461 11484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } }
I20260923 17:40:24.368726 11484 leader_election.cc:288] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [CANDIDATE]: Term 1 election: Requested vote from peers 62e679a38a28496a87d9ac7455854869 (127.20.240.125:44647), c3034379655644cab95402e0098fffa4 (127.20.240.124:37957)
I20260923 17:40:24.369429 11633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "948b3642de9b4df4a75074b88ea94d9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c3034379655644cab95402e0098fffa4"
I20260923 17:40:24.369961 11633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:24.373906 11633 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b3642de9b4df4a75074b88ea94d9b in term 1.
I20260923 17:40:24.374693 11458 leader_election.cc:302] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 948b3642de9b4df4a75074b88ea94d9b, c3034379655644cab95402e0098fffa4; no voters:
I20260923 17:40:24.375272 11678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:40:24.385129 11678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [term 1 LEADER]: Becoming Leader. State: Replica: 948b3642de9b4df4a75074b88ea94d9b, State: Running, Role: LEADER
I20260923 17:40:24.386080 11678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } }
I20260923 17:40:24.396837 11687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 948b3642de9b4df4a75074b88ea94d9b. Latest consensus state: current_term: 1 leader_uuid: "948b3642de9b4df4a75074b88ea94d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } } }
I20260923 17:40:24.397377 11687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:24.414503 11696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:40:24.415006 11584 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } attempt: 5
W20260923 17:40:24.416519 11463 tablet.cc:2369] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:40:24.418115 11699 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:40:24.435482 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.397415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.20.240.126:45519)
0923 17:40:24.397848 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.397855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.415578 (+ 17723us) server_negotiation.cc:407] Connection header received
0923 17:40:24.415923 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.415929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.416058 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.416134 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.419444 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.420539 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.422746 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.423533 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.423620 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.427199 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:24.427210 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:24.427215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:24.427268 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:24.431500 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.432454 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.432460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.432463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.432678 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.433160 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.433165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.433169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.433587 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:24.433854 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:24.434714 (+ 860us) server_negotiation.cc:299] Negotiation successful
0923 17:40:24.435074 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288}
I20260923 17:40:24.435920 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.403097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.20.240.124:37957)
0923 17:40:24.403320 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.403326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.416057 (+ 12731us) server_negotiation.cc:407] Connection header received
0923 17:40:24.416613 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.416619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.416729 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.416819 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.419451 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.420568 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.422747 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.423480 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.423602 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.428202 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:24.428210 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:24.428215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:24.428260 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:24.431804 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.433045 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.433052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.433055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.433228 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.433896 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.433901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.433904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.434226 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:24.434470 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:24.435259 (+ 789us) server_negotiation.cc:299] Negotiation successful
0923 17:40:24.435498 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260923 17:40:24.426321 11442 master_runner.cc:186] Error getting master registration for 127.20.240.125:44647: OK, instance_id { permanent_uuid: "62e679a38a28496a87d9ac7455854869" instance_seqno: 1790185223684417 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:40:24.437839 11442 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:40:24.440582 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.397630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.20.240.125:44647)
0923 17:40:24.398000 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.398006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.405860 (+ 7854us) server_negotiation.cc:407] Connection header received
0923 17:40:24.406483 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.406490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.406626 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.406712 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.409381 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.410544 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.413140 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.413822 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.413936 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.418011 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:24.418027 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:24.418032 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:24.418085 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:24.421841 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.438272 (+ 16431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.438278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.438283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.438455 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.438854 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.438858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.438862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.439322 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:24.439577 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:24.439939 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 17:40:24.440228 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
W20260923 17:40:24.446645 11612 tablet.cc:2369] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:40:24.450460 11696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 17:40:24 dist-test-slave-ppjl krb5kdc[11429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185223, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.125@KRBTEST.COM for kudu/127.20.240.124@KRBTEST.COM
I20260923 17:40:24.463869 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.415656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:37957 (local address 127.0.0.1:55004)
0923 17:40:24.416045 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:24.416063 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:24.416173 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:24.436455 (+ 20282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:24.436461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:24.436978 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:24.437301 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:24.437317 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.447959 (+ 10642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.447966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:24.448963 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:24.448982 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.450834 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.450840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:24.451075 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.451832 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:24.451857 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:24.454553 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:24.460706 (+ 6153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:24.460712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:24.460717 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:24.461159 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:24.461767 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:24.461773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:24.461777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:24.461994 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:24.462715 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:24.462721 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:24.462890 (+ 169us) client_negotiation.cc:769] Sending connection context
0923 17:40:24.463034 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 17:40:24.463440 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269}
I20260923 17:40:24.465917 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.416411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55004 (local address 127.20.240.124:37957)
0923 17:40:24.436147 (+ 19736us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.436152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.436168 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:40:24.436211 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.436216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.436285 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.436380 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.437494 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.447782 (+ 10288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.450056 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.450651 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.453333 (+ 2682us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.454801 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:24.454808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:24.454813 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:24.454854 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:24.460564 (+ 5710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.461359 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.461366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.461369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.461593 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:24.462119 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:24.462131 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:24.462135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:24.462535 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:24.462781 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:24.463218 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 17:40:24.463457 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19625,"thread_start_us":31974,"threads_started":1}
I20260923 17:40:24.470671 11633 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4 [term 1 FOLLOWER]: Refusing update from remote peer 948b3642de9b4df4a75074b88ea94d9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:40:24.471798 11584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869: Bootstrap starting.
I20260923 17:40:24.472085 11687 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:40:24.477734 11584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:24.479220 11584 log.cc:824] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:24.489154 11584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869: No bootstrap required, opened a new log
I20260923 17:40:24.500610 11584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } }
I20260923 17:40:24.501145 11584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62e679a38a28496a87d9ac7455854869, State: Initialized, Role: FOLLOWER
I20260923 17:40:24.501767 11584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } }
I20260923 17:40:24.509109 11664 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 948b3642de9b4df4a75074b88ea94d9b. Latest consensus state: current_term: 1 leader_uuid: "948b3642de9b4df4a75074b88ea94d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } } }
I20260923 17:40:24.509511 11664 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:40:24.522960 11706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } } }
I20260923 17:40:24.523825 11706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:40:24.527266 11558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "948b3642de9b4df4a75074b88ea94d9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "62e679a38a28496a87d9ac7455854869"
I20260923 17:40:24.528146 11557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:24.528752 11584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:24.533185 11664 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "948b3642de9b4df4a75074b88ea94d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } } }
I20260923 17:40:24.533460 11664 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:40:24.534770 11557 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869 [term 1 FOLLOWER]: Refusing update from remote peer 948b3642de9b4df4a75074b88ea94d9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:40:24.537592 11687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "948b3642de9b4df4a75074b88ea94d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } } }
I20260923 17:40:24.537881 11687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:24.538317 11558 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 948b3642de9b4df4a75074b88ea94d9b in term 1.
I20260923 17:40:24.540738 11696 catalog_manager.cc:1347] Generated new cluster ID: e20ea83dbaee4e118602164b79fb27b7
I20260923 17:40:24.540856 11696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:40:24.540946 11678 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:40:24.554661 11696 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:40:24.557024 11696 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:40:24.585391 11696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b: Generated new TSK 0
I20260923 17:40:24.586570 11696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:40:24.589681 11717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "948b3642de9b4df4a75074b88ea94d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } } }
I20260923 17:40:24.590145 11717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:40:24.589990 11706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 948b3642de9b4df4a75074b88ea94d9b. Latest consensus state: current_term: 1 leader_uuid: "948b3642de9b4df4a75074b88ea94d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } } }
I20260923 17:40:24.590287 11706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:40:24.591724 11687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "948b3642de9b4df4a75074b88ea94d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "948b3642de9b4df4a75074b88ea94d9b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 45519 } } peers { permanent_uuid: "62e679a38a28496a87d9ac7455854869" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 44647 } } peers { permanent_uuid: "c3034379655644cab95402e0098fffa4" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 37957 } } }
I20260923 17:40:24.592001 11687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 948b3642de9b4df4a75074b88ea94d9b [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:24.622216 11725 catalog_manager.cc:1259] Loaded cluster ID: e20ea83dbaee4e118602164b79fb27b7
I20260923 17:40:24.622403 11725 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869: loading cluster ID for follower catalog manager: success
I20260923 17:40:24.626327 11725 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869: acquiring CA information for follower catalog manager: success
I20260923 17:40:24.629056 11725 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 62e679a38a28496a87d9ac7455854869: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 17:40:24.726038 11588 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:24.712038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.20.240.125:44647)
0923 17:40:24.712257 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.712263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.723806 (+ 11543us) server_negotiation.cc:407] Connection header received
0923 17:40:24.724124 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.724131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.724226 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.724310 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.725720 (+ 1410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40354: BlockingRecv error: recv got EOF from 127.0.0.1:40354 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260923 17:40:24.731031 11512 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:24.724840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.20.240.126:45519)
0923 17:40:24.725064 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.725070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.728860 (+ 3790us) server_negotiation.cc:407] Connection header received
0923 17:40:24.729484 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.729491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.729578 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.729649 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.730777 (+ 1128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34330: BlockingRecv error: recv got EOF from 127.0.0.1:34330 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 17:40:24.740316 11661 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:24.718366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55016 (local address 127.20.240.124:37957)
0923 17:40:24.718579 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.718585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.726846 (+ 8261us) server_negotiation.cc:407] Connection header received
0923 17:40:24.738785 (+ 11939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.738792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.738882 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.738956 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.740058 (+ 1102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55016: BlockingRecv error: recv got EOF from 127.0.0.1:55016 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 17:40:24.794121 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.773621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40366 (local address 127.20.240.125:44647)
0923 17:40:24.773806 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.773811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.785342 (+ 11531us) server_negotiation.cc:407] Connection header received
0923 17:40:24.785989 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.785996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.786266 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.786420 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:24.787266 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.788525 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.790815 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.791471 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.791636 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.792485 (+ 849us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:24.793284 (+ 799us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:24.793407 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:24.793670 (+ 263us) server_negotiation.cc:299] Negotiation successful
0923 17:40:24.793777 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 17:40:24.842299 11661 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:24.828606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55018 (local address 127.20.240.124:37957)
0923 17:40:24.828836 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.828842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.839915 (+ 11073us) server_negotiation.cc:407] Connection header received
0923 17:40:24.840565 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.840573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.840709 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.840802 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.842019 (+ 1217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55018: BlockingRecv error: recv got EOF from 127.0.0.1:55018 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:40:24.898492 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.879649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40376 (local address 127.20.240.125:44647)
0923 17:40:24.879887 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.879892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.890634 (+ 10742us) server_negotiation.cc:407] Connection header received
0923 17:40:24.891286 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.891293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.891445 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.891566 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:24.892732 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.893925 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.896029 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.896781 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.896892 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.897565 (+ 673us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:24.897704 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:24.897771 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:24.898028 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 17:40:24.898147 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260923 17:40:24.949688 11661 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:24.941780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.20.240.124:37957)
0923 17:40:24.941974 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.941980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.947896 (+ 5916us) server_negotiation.cc:407] Connection header received
0923 17:40:24.948182 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.948188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.948298 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.948442 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.949453 (+ 1011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55030: BlockingRecv error: recv got EOF from 127.0.0.1:55030 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:40:24.964403 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.956587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40380 (local address 127.20.240.125:44647)
0923 17:40:24.956780 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.956787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.956815 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:40:24.957159 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.957166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.957267 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.957393 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:24.957872 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.959085 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.961561 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.962280 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.962442 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.963280 (+ 838us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:24.963484 (+ 204us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:24.963556 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:24.963943 (+ 387us) server_negotiation.cc:299] Negotiation successful
0923 17:40:24.964046 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260923 17:40:24.975543 11661 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:24.973553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55038 (local address 127.20.240.124:37957)
0923 17:40:24.973757 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.973763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.973792 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:40:24.974144 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.974151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.974253 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.974369 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:24.975297 (+ 928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55038: BlockingRecv error: recv got EOF from 127.0.0.1:55038 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 17:40:24.993809 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:24.983464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.20.240.125:44647)
0923 17:40:24.983712 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:24.983718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:24.984277 (+ 559us) server_negotiation.cc:407] Connection header received
0923 17:40:24.984507 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:24.984513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:24.984607 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:24.984696 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:24.985226 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.986408 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.988491 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:24.989167 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:24.989274 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:24.990466 (+ 1192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:24.990673 (+ 207us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:24.993024 (+ 2351us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:24.993350 (+ 326us) server_negotiation.cc:299] Negotiation successful
0923 17:40:24.993462 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
W20260923 17:40:25.007547 11661 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:25.005224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.20.240.124:37957)
0923 17:40:25.005500 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:25.005506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:25.005998 (+ 492us) server_negotiation.cc:407] Connection header received
0923 17:40:25.006187 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:25.006193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:25.006287 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:25.006390 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:25.007306 (+ 916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55050: BlockingRecv error: recv got EOF from 127.0.0.1:55050 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
I20260923 17:40:25.027691 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:25.020924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40400 (local address 127.20.240.125:44647)
0923 17:40:25.021110 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:25.021116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:25.021708 (+ 592us) server_negotiation.cc:407] Connection header received
0923 17:40:25.021923 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:25.021929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:25.022026 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:25.022136 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:25.022507 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.023766 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.025327 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.025973 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.026132 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:25.026789 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:25.026967 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:25.027080 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:25.027270 (+ 190us) server_negotiation.cc:299] Negotiation successful
0923 17:40:25.027365 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260923 17:40:25.036037 11661 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:25.034021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.20.240.124:37957)
0923 17:40:25.034207 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:25.034214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:25.034585 (+ 371us) server_negotiation.cc:407] Connection header received
0923 17:40:25.034779 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:25.034783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:25.034859 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:25.034949 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:25.035747 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55052: BlockingRecv error: recv got EOF from 127.0.0.1:55052 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:40:25.062172 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:25.055823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.20.240.125:44647)
0923 17:40:25.056052 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:25.056058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:25.056438 (+ 380us) server_negotiation.cc:407] Connection header received
0923 17:40:25.056614 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:25.056619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:25.056722 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:25.056898 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:25.057296 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.058498 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.060024 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.060732 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.060887 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:25.061363 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:25.061511 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:25.061596 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:25.061759 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 17:40:25.061871 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260923 17:40:25.076944 11661 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:25.074445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.20.240.124:37957)
0923 17:40:25.074694 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:25.074701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:25.075116 (+ 415us) server_negotiation.cc:407] Connection header received
0923 17:40:25.075308 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:25.075315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:25.075415 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:25.075554 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:25.076691 (+ 1137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55062: BlockingRecv error: recv got EOF from 127.0.0.1:55062 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260923 17:40:25.119360 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:25.113190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40410 (local address 127.20.240.125:44647)
0923 17:40:25.113431 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:25.113437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:25.113793 (+ 356us) server_negotiation.cc:407] Connection header received
0923 17:40:25.113973 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:25.113979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:25.114066 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:25.114161 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:25.114597 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.115717 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.117232 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.117917 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.118046 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:25.118484 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:25.118619 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:25.118705 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:25.118917 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 17:40:25.119005 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260923 17:40:25.127640 11661 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:25.125620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.20.240.124:37957)
0923 17:40:25.125815 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:25.125822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:25.126172 (+ 350us) server_negotiation.cc:407] Connection header received
0923 17:40:25.126355 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:25.126360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:25.126479 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:25.126590 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:25.127408 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55070: BlockingRecv error: recv got EOF from 127.0.0.1:55070 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:40:25.200344 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:25.195256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40424 (local address 127.20.240.125:44647)
0923 17:40:25.195446 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:25.195453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:25.195805 (+ 352us) server_negotiation.cc:407] Connection header received
0923 17:40:25.196006 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:25.196010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:25.196083 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:25.196162 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:25.196574 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.197300 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.198649 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.199087 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.199241 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:25.199639 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:25.199757 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:25.199836 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:25.199970 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 17:40:25.200056 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260923 17:40:25.208109 11661 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:25.206247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.20.240.124:37957)
0923 17:40:25.206455 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:25.206462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:25.206766 (+ 304us) server_negotiation.cc:407] Connection header received
0923 17:40:25.206956 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:25.206961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:25.207036 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:25.207121 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:25.207892 (+ 771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55072: BlockingRecv error: recv got EOF from 127.0.0.1:55072 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:40:25.311491 11675 catalog_manager.cc:1259] Loaded cluster ID: e20ea83dbaee4e118602164b79fb27b7
I20260923 17:40:25.311611 11675 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4: loading cluster ID for follower catalog manager: success
I20260923 17:40:25.314270 11675 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4: acquiring CA information for follower catalog manager: success
I20260923 17:40:25.316284 11675 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c3034379655644cab95402e0098fffa4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:40:25.344722 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:25.339739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.20.240.125:44647)
0923 17:40:25.339942 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:25.339947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:25.340226 (+ 279us) server_negotiation.cc:407] Connection header received
0923 17:40:25.340396 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:25.340401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:25.340494 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:25.340587 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:25.341022 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.341822 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.343255 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.343710 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.343815 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:25.344182 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:25.344290 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:25.344353 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:25.344448 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 17:40:25.344512 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:40:25.355814 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:25.350252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55082 (local address 127.20.240.124:37957)
0923 17:40:25.350452 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:25.350459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:25.350712 (+ 253us) server_negotiation.cc:407] Connection header received
0923 17:40:25.350906 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:25.350911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:25.351083 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:25.351209 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:25.351646 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.352365 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.353632 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.354034 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.354143 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:25.354655 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:25.355199 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:25.355330 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:25.355430 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 17:40:25.355526 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:40:25.371122 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:25.362999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.20.240.125:44647)
0923 17:40:25.363241 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:25.363248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:25.364650 (+ 1402us) server_negotiation.cc:407] Connection header received
0923 17:40:25.364900 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:25.364905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:25.364987 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:25.365097 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:25.365866 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.366956 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.369115 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.369624 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.369755 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:25.370225 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:25.370393 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:25.370490 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:25.370708 (+ 218us) server_negotiation.cc:299] Negotiation successful
0923 17:40:25.370802 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:40:25.371905 11830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:25.363082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.20.240.126:45519)
0923 17:40:25.363636 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:25.363642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:25.364568 (+ 926us) server_negotiation.cc:407] Connection header received
0923 17:40:25.364849 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:25.364855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:25.365090 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:25.365262 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:25.366103 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.367124 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.369257 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.369675 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.369780 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:25.370408 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:25.370911 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:25.371018 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:25.371221 (+ 203us) server_negotiation.cc:299] Negotiation successful
0923 17:40:25.371569 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":417,"threads_started":1}
I20260923 17:40:25.372169 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:25.363443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.20.240.124:37957)
0923 17:40:25.363619 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:25.363626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:25.365739 (+ 2113us) server_negotiation.cc:407] Connection header received
0923 17:40:25.365982 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:25.365988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:25.366087 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:25.366177 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:25.366711 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.367703 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.370414 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:25.370857 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:25.370972 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:25.371493 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:25.371589 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:25.371636 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:25.371812 (+ 176us) server_negotiation.cc:299] Negotiation successful
0923 17:40:25.371891 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 17:40:25.377144 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 11442
I20260923 17:40:25.378336 11504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:40:25.488232 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 11442
W20260923 17:40:25.506124 11609 connection.cc:535] server connection from 127.0.0.1:54998 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 17:40:25.506762 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 11516
I20260923 17:40:25.507849 11579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:40:25.614193 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 11516
I20260923 17:40:25.631078 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 11591
I20260923 17:40:25.632236 11653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:40:25.739343 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 11591
2026-09-23T17:40:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4279 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:40:25 dist-test-slave-ppjl krb5kdc[11839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:40:25 dist-test-slave-ppjl krb5kdc[11839](info): set up 2 sockets
Sep 23 17:40:25 dist-test-slave-ppjl krb5kdc[11839](info): commencing operation
krb5kdc: starting...
W20260923 17:40:26.772307 21441 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/test-user.keytab.
Sep 23 17:40:26 dist-test-slave-ppjl krb5kdc[11839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:40:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:40:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
Entry for principal kudu/127.20.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab.
I20260923 17:40:27.027698 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:35711
--webserver_interface=127.20.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:35711,127.20.240.125:46573,127.20.240.124:36689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:40:27.363704 11852 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:40:27.364069 11852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:27.364174 11852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:27.364285 11852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:40:27.372267 11852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:27.372380 11852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:27.372442 11852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:27.372486 11852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:27.372557 11852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:27.382973 11852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:35711,127.20.240.125:46573,127.20.240.124:36689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.126.keytab
--principal=kudu/127.20.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:35711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:27.384338 11852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:27.385689 11852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:27.397789 11857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:27.397891 11858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:27.398006 11860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:27.398602 11852 server_base.cc:1034] running on GCE node
I20260923 17:40:27.399472 11852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:27.400986 11852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:27.402199 11852 hybrid_clock.cc:648] HybridClock initialized: now 1790185227402129 us; error 42 us; skew 500 ppm
I20260923 17:40:27.402618 11852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:27 dist-test-slave-ppjl krb5kdc[11839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185227, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:27.410991 11852 init.cc:377] Logged in from keytab as kudu/127.20.240.126@KRBTEST.COM (short username kudu)
I20260923 17:40:27.412643 11852 webserver.cc:460] Webserver started at http://127.20.240.126:40719/ using document root <none> and password file <none>
I20260923 17:40:27.413380 11852 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:27.413477 11852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:27.413815 11852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:27.417328 11852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/data/instance:
uuid: "72aa1c828d004f10bad438fb32e0cb54"
format_stamp: "Formatted at 2026-09-23 17:40:27 on dist-test-slave-ppjl"
server_key: "aed8e4616f1eb07d87990e9d6865accf"
server_key_iv: "33e5503fb422662ce7b28c1cf195ec96"
server_key_version: "encryptionkey@0"
I20260923 17:40:27.418237 11852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/wal/instance:
uuid: "72aa1c828d004f10bad438fb32e0cb54"
format_stamp: "Formatted at 2026-09-23 17:40:27 on dist-test-slave-ppjl"
server_key: "aed8e4616f1eb07d87990e9d6865accf"
server_key_iv: "33e5503fb422662ce7b28c1cf195ec96"
server_key_version: "encryptionkey@0"
I20260923 17:40:27.423839 11852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:40:27.427457 11867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:27.428942 11852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:40:27.429126 11852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/wal
uuid: "72aa1c828d004f10bad438fb32e0cb54"
format_stamp: "Formatted at 2026-09-23 17:40:27 on dist-test-slave-ppjl"
server_key: "aed8e4616f1eb07d87990e9d6865accf"
server_key_iv: "33e5503fb422662ce7b28c1cf195ec96"
server_key_version: "encryptionkey@0"
I20260923 17:40:27.429339 11852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:27.456115 11852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:27.460563 11852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:27.460826 11852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:27.479198 11852 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:35711
I20260923 17:40:27.479251 11918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:35711 every 8 connection(s)
I20260923 17:40:27.480682 11852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/data/info.pb
I20260923 17:40:27.484047 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 11852
I20260923 17:40:27.484233 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-0/wal/instance
I20260923 17:40:27.484792 21441 external_mini_cluster.cc:1428] Setting key 84f2ce4b45349a57adb324b7424f86e5
I20260923 17:40:27.484933 11919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:27.493940 11919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:40:27 dist-test-slave-ppjl krb5kdc[11839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185227, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.240.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.125@KRBTEST.COM" created.
I20260923 17:40:27.518710 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:27.497716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35711 (local address 127.0.0.1:56524)
0923 17:40:27.498568 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:27.498624 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:27.500748 (+ 2124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:27.501750 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:27.501755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:27.502315 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:27.502686 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:27.502705 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:27.504780 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:27.504784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:27.505901 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:27.505913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:27.507075 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:27.507080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:27.507306 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:27.507935 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:27.507958 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:27.510339 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:27.514126 (+ 3787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:27.514138 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:27.514149 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:27.514791 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:27.515440 (+ 649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:27.515444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:27.515447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:27.515613 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:27.516607 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:27.516614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:27.516819 (+ 205us) client_negotiation.cc:769] Sending connection context
0923 17:40:27.517046 (+ 227us) client_negotiation.cc:240] Negotiation successful
0923 17:40:27.517513 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":1,"thread_start_us":344,"threads_started":1}
I20260923 17:40:27.518765 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:27.499056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56524 (local address 127.20.240.126:35711)
0923 17:40:27.499777 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:27.499785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:27.499841 (+ 56us) server_negotiation.cc:407] Connection header received
0923 17:40:27.501150 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:27.501157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:27.501479 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:27.501710 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:27.502912 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:27.504614 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:27.506047 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:27.506847 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:27.507041 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:27.510690 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:27.510702 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:27.510721 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:27.510761 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:27.513985 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:27.514964 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:27.514974 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:27.514977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:27.515272 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:27.515723 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:27.515728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:27.515732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:27.516443 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:27.516640 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:27.517220 (+ 580us) server_negotiation.cc:299] Negotiation successful
0923 17:40:27.517659 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":254,"server-negotiator.queue_time_us":523,"thread_start_us":357,"threads_started":1}
I20260923 17:40:27.522377 11919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:40:27.523617 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:27.523109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:46573 (local address 127.0.0.1:53108)
0923 17:40:27.523327 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:27.523461 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260923 17:40:27.523890 11870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
I20260923 17:40:27.525599 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:27.525175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:46573 (local address 127.0.0.1:53118)
0923 17:40:27.525352 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:27.525437 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260923 17:40:27.525821 11919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.125:46573: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.125.keytab.
Entry for principal kudu/127.20.240.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.125.keytab.
I20260923 17:40:27.546504 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.125:46573
--webserver_interface=127.20.240.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.125.keytab
--principal=kudu/127.20.240.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:35711,127.20.240.125:46573,127.20.240.124:36689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:40:27.575263 11919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } attempt: 1
I20260923 17:40:27.576625 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:27.576043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:46573 (local address 127.0.0.1:53126)
0923 17:40:27.576258 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:27.576477 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260923 17:40:27.577757 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:27.577306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:46573 (local address 127.0.0.1:53130)
0923 17:40:27.577470 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:27.577615 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 17:40:27.577939 11919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.125:46573: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
I20260923 17:40:27.646226 11919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } attempt: 2
I20260923 17:40:27.647619 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:27.646972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:46573 (local address 127.0.0.1:53140)
0923 17:40:27.647297 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:27.647394 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 17:40:27.648630 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:27.648253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:46573 (local address 127.0.0.1:53154)
0923 17:40:27.648412 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:27.648500 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 17:40:27.648808 11919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.125:46573: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
I20260923 17:40:27.740082 11919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } attempt: 3
I20260923 17:40:27.741381 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:27.740868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:46573 (local address 127.0.0.1:53170)
0923 17:40:27.741127 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:27.741240 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260923 17:40:27.742434 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:27.742069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:46573 (local address 127.0.0.1:53174)
0923 17:40:27.742215 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:27.742288 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 17:40:27.742635 11919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.125:46573: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
W20260923 17:40:27.878543 11926 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:40:27.878934 11926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:27.879033 11926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:27.879146 11926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 17:40:27.885910 11919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } attempt: 4
W20260923 17:40:27.886955 11926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:27.887050 11926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:27.887141 11926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:27.887230 11926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:27.887316 11926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:27.887319 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:27.886646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:46573 (local address 127.0.0.1:53184)
0923 17:40:27.887007 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:27.887112 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260923 17:40:27.888299 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:27.887961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:46573 (local address 127.0.0.1:53194)
0923 17:40:27.888092 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:27.888181 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 17:40:27.888487 11919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.125:46573: Network error: Client connection negotiation failed: client connection to 127.20.240.125:46573: connect: Connection refused (error 111)
I20260923 17:40:27.898056 11926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:35711,127.20.240.125:46573,127.20.240.124:36689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.125.keytab
--principal=kudu/127.20.240.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.125:46573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.240.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:27.899542 11926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:27.900909 11926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:27.912839 11932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:27.912881 11934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:27.912894 11931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:27.913758 11926 server_base.cc:1034] running on GCE node
I20260923 17:40:27.914634 11926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:27.916241 11926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:27.917505 11926 hybrid_clock.cc:648] HybridClock initialized: now 1790185227917423 us; error 54 us; skew 500 ppm
I20260923 17:40:27.917871 11926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:27 dist-test-slave-ppjl krb5kdc[11839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185227, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:27.921355 11926 init.cc:377] Logged in from keytab as kudu/127.20.240.125@KRBTEST.COM (short username kudu)
I20260923 17:40:27.923125 11926 webserver.cc:460] Webserver started at http://127.20.240.125:42337/ using document root <none> and password file <none>
I20260923 17:40:27.923976 11926 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:27.924101 11926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:27.924599 11926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:27.927969 11926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/data/instance:
uuid: "b9256a28a7aa43679ddece9936d38503"
format_stamp: "Formatted at 2026-09-23 17:40:27 on dist-test-slave-ppjl"
server_key: "344c5df329118614eed0554ff20ad873"
server_key_iv: "368a931b65df2c953b4bafe15b24f848"
server_key_version: "encryptionkey@0"
I20260923 17:40:27.928795 11926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/wal/instance:
uuid: "b9256a28a7aa43679ddece9936d38503"
format_stamp: "Formatted at 2026-09-23 17:40:27 on dist-test-slave-ppjl"
server_key: "344c5df329118614eed0554ff20ad873"
server_key_iv: "368a931b65df2c953b4bafe15b24f848"
server_key_version: "encryptionkey@0"
I20260923 17:40:27.934300 11926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:40:27.938122 11941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:27.939646 11926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:40:27.939822 11926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/wal
uuid: "b9256a28a7aa43679ddece9936d38503"
format_stamp: "Formatted at 2026-09-23 17:40:27 on dist-test-slave-ppjl"
server_key: "344c5df329118614eed0554ff20ad873"
server_key_iv: "368a931b65df2c953b4bafe15b24f848"
server_key_version: "encryptionkey@0"
I20260923 17:40:27.940033 11926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:27.963965 11926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:27.968343 11926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:27.968611 11926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:27.986304 11926 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.125:46573
I20260923 17:40:27.986328 11993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.125:46573 every 8 connection(s)
I20260923 17:40:27.987677 11926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/data/info.pb
I20260923 17:40:27.991581 11994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:27.992002 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 11926
I20260923 17:40:27.992158 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-1/wal/instance
I20260923 17:40:27.992630 21441 external_mini_cluster.cc:1428] Setting key 1e6677d9033bac3ec4fa7f65d820f259
I20260923 17:40:27.998418 11994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:40:28 dist-test-slave-ppjl krb5kdc[11839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185227, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.125@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.240.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.240.124@KRBTEST.COM" created.
I20260923 17:40:28.018608 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.001204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56532 (local address 127.20.240.126:35711)
0923 17:40:28.001383 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:28.001388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:28.002636 (+ 1248us) server_negotiation.cc:407] Connection header received
0923 17:40:28.004718 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:28.004723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:28.004818 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:28.004905 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:28.006532 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.007264 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.008832 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.009242 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.009354 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.012772 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:28.012777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:28.012781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:28.012814 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:28.015519 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.016252 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.016256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.016259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.016377 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.016788 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.016793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.016796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.017106 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:28.017264 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:28.018034 (+ 770us) server_negotiation.cc:299] Negotiation successful
0923 17:40:28.018287 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:40:28.019483 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.001715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35711 (local address 127.0.0.1:56532)
0923 17:40:28.002469 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.002508 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:28.004419 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:28.005177 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:28.005186 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:28.005737 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:28.006316 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.006334 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.007428 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.007434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.008673 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.008686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.009468 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.009476 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.009707 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.010360 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:28.010383 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:28.012549 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:28.015668 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.015678 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.015697 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.016134 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.016489 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.016495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.016499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.016684 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.017241 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:28.017248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:28.017634 (+ 386us) client_negotiation.cc:769] Sending connection context
0923 17:40:28.017867 (+ 233us) client_negotiation.cc:240] Negotiation successful
0923 17:40:28.018309 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":296,"threads_started":1}
I20260923 17:40:28.021345 11994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:40:28 dist-test-slave-ppjl krb5kdc[11839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185227, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.125@KRBTEST.COM for kudu/127.20.240.125@KRBTEST.COM
I20260923 17:40:28.036496 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.021989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:46573 (local address 127.0.0.1:53202)
0923 17:40:28.022166 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.022184 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:28.022279 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:28.023188 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:28.023192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:28.023616 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:28.023778 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.023789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.025110 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.025115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.026029 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.026040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.027101 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.027105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.027326 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.027883 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:28.027900 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:28.030027 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:28.033945 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.033952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.033956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.034402 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.034802 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.034805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.034807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.034934 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.035691 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:28.035695 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:28.035819 (+ 124us) client_negotiation.cc:769] Sending connection context
0923 17:40:28.035927 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 17:40:28.036176 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 17:40:28.036763 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.022201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.20.240.125:46573)
0923 17:40:28.022748 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:28.022754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:28.022779 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:40:28.022825 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:28.022831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:28.023020 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:28.023134 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:28.023936 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.024974 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.026162 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.026956 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.027072 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.030253 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:28.030269 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:28.030289 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:28.030323 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:28.033758 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.034534 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.034542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.034545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.034696 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.035062 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.035066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.035070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.035564 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:28.035756 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:28.036076 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 17:40:28.036319 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":302,"threads_started":1}
I20260923 17:40:28.039518 11994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:40:28.040858 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.040280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45242)
0923 17:40:28.040488 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.040645 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260923 17:40:28.041181 11942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
I20260923 17:40:28.042824 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.042380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45244)
0923 17:40:28.042580 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.042682 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 17:40:28.043047 11994 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:36689: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.240.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.124.keytab.
Entry for principal kudu/127.20.240.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.124.keytab.
I20260923 17:40:28.051424 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
/tmp/dist-test-taskSmAN3S/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.124:36689
--webserver_interface=127.20.240.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.124.keytab
--principal=kudu/127.20.240.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.240.84:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:35711,127.20.240.125:46573,127.20.240.124:36689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:40:28.092417 11994 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } attempt: 1
I20260923 17:40:28.093734 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.093174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45260)
0923 17:40:28.093403 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.093518 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 17:40:28.094766 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.094374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45266)
0923 17:40:28.094536 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.094633 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 17:40:28.095032 11994 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:36689: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
I20260923 17:40:28.163285 11994 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } attempt: 2
I20260923 17:40:28.164726 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.164073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45282)
0923 17:40:28.164301 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.164531 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 17:40:28.165818 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.165407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45286)
0923 17:40:28.165568 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.165662 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 17:40:28.166054 11994 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:36689: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
I20260923 17:40:28.187722 11919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } attempt: 5
Sep 23 17:40:28 dist-test-slave-ppjl krb5kdc[11839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185227, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for kudu/127.20.240.125@KRBTEST.COM
I20260923 17:40:28.203408 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.188737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:46573 (local address 127.0.0.1:53208)
0923 17:40:28.188944 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.188964 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:28.189099 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:28.189710 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:28.189716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:28.190210 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:28.190493 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.190508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.191530 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.191536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.192859 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.192875 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.193737 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.193744 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.193975 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.194554 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:28.194577 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:28.197052 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:28.200914 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.200919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.200923 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.201326 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.201774 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.201778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.201781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.201916 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.202638 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:28.202643 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:28.202777 (+ 134us) client_negotiation.cc:769] Sending connection context
0923 17:40:28.202899 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 17:40:28.203125 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 17:40:28.203616 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.188718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53208 (local address 127.20.240.125:46573)
0923 17:40:28.189344 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:28.189350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:28.189370 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:40:28.189431 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:28.189436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:28.189540 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:28.189634 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:28.190650 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.191393 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.193017 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.193554 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.193651 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.197294 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:28.197301 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:28.197307 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:28.197365 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:28.200734 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.201460 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.201466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.201470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.201643 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.202037 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.202042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.202046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.202471 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:28.202636 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:28.203027 (+ 391us) server_negotiation.cc:299] Negotiation successful
0923 17:40:28.203297 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503}
I20260923 17:40:28.204609 11919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:40:28.205547 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.205183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45290)
0923 17:40:28.205342 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.205421 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 17:40:28.206431 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.206150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45294)
0923 17:40:28.206274 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.206336 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 17:40:28.206657 11919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:36689: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
I20260923 17:40:28.257336 11994 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } attempt: 3
I20260923 17:40:28.257879 11919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } attempt: 1
I20260923 17:40:28.258975 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.258460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45302)
0923 17:40:28.258680 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.258785 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260923 17:40:28.258987 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.258093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45300)
0923 17:40:28.258710 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.258795 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260923 17:40:28.260092 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.259672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45308)
0923 17:40:28.259846 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.259947 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 17:40:28.260377 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.259920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45314)
0923 17:40:28.260141 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.260235 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260923 17:40:28.260380 11919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:36689: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
W20260923 17:40:28.260600 11994 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:36689: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
I20260923 17:40:28.328642 11919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } attempt: 2
I20260923 17:40:28.330041 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.329409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45316)
0923 17:40:28.329671 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.329802 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260923 17:40:28.331205 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.330731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45326)
0923 17:40:28.330910 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.331030 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 17:40:28.331470 11919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:36689: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
W20260923 17:40:28.391687 12001 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:40:28.392022 12001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:40:28.392118 12001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:40:28.392210 12001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:40:28.399572 12001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:40:28.399664 12001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:40:28.399722 12001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:40:28.399791 12001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:40:28.399878 12001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:40:28.403791 11994 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } attempt: 4
I20260923 17:40:28.404964 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.404493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45340)
0923 17:40:28.404716 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.404812 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 17:40:28.405905 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.405586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45342)
0923 17:40:28.405733 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.405799 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:40:28.406116 11994 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:36689: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
I20260923 17:40:28.410346 12001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:35711,127.20.240.125:46573,127.20.240.124:36689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/krb5kdc/kudu_127.20.240.124.keytab
--principal=kudu/127.20.240.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.124:36689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.240.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:40:28.411695 12001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:40:28.413057 12001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:40:28.424707 12006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:28.424716 12007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:40:28.424692 12009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:40:28.425776 12001 server_base.cc:1034] running on GCE node
I20260923 17:40:28.426535 12001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:40:28.428058 12001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:40:28.429277 12001 hybrid_clock.cc:648] HybridClock initialized: now 1790185228429209 us; error 44 us; skew 500 ppm
I20260923 17:40:28.429611 12001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:40:28 dist-test-slave-ppjl krb5kdc[11839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185228, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:40:28.432842 12001 init.cc:377] Logged in from keytab as kudu/127.20.240.124@KRBTEST.COM (short username kudu)
I20260923 17:40:28.434461 12001 webserver.cc:460] Webserver started at http://127.20.240.124:45343/ using document root <none> and password file <none>
I20260923 17:40:28.435209 12001 fs_manager.cc:359] Metadata directory not provided
I20260923 17:40:28.435315 12001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:40:28.435621 12001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:40:28.437695 11919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } attempt: 3
I20260923 17:40:28.438687 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.438224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45350)
0923 17:40:28.438435 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.438518 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 17:40:28.438679 12001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/data/instance:
uuid: "aee89b2883f94d05b915688995be633d"
format_stamp: "Formatted at 2026-09-23 17:40:28 on dist-test-slave-ppjl"
server_key: "af39fd7635b82e4b6613732ee5a168b3"
server_key_iv: "0f45a50e2028974a49b437984f65b2a7"
server_key_version: "encryptionkey@0"
I20260923 17:40:28.439726 12001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/wal/instance:
uuid: "aee89b2883f94d05b915688995be633d"
format_stamp: "Formatted at 2026-09-23 17:40:28 on dist-test-slave-ppjl"
server_key: "af39fd7635b82e4b6613732ee5a168b3"
server_key_iv: "0f45a50e2028974a49b437984f65b2a7"
server_key_version: "encryptionkey@0"
I20260923 17:40:28.439926 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.439449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45366)
0923 17:40:28.439697 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.439794 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260923 17:40:28.440125 11919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.240.124:36689: Network error: Client connection negotiation failed: client connection to 127.20.240.124:36689: connect: Connection refused (error 111)
I20260923 17:40:28.445055 12001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 17:40:28.448586 12016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:40:28.449918 12001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:40:28.450089 12001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/data,/tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/wal
uuid: "aee89b2883f94d05b915688995be633d"
format_stamp: "Formatted at 2026-09-23 17:40:28 on dist-test-slave-ppjl"
server_key: "af39fd7635b82e4b6613732ee5a168b3"
server_key_iv: "0f45a50e2028974a49b437984f65b2a7"
server_key_version: "encryptionkey@0"
I20260923 17:40:28.450286 12001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:40:28.470068 12001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:40:28.474357 12001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:40:28.474642 12001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:40:28.493115 12001 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.124:36689
I20260923 17:40:28.493134 12067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.124:36689 every 8 connection(s)
I20260923 17:40:28.494513 12001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/data/info.pb
I20260923 17:40:28.496366 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu as pid 12001
I20260923 17:40:28.496572 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSmAN3S/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184865977475-21441-0/minicluster-data/master-2/wal/instance
I20260923 17:40:28.497169 21441 external_mini_cluster.cc:1428] Setting key 8513d75c1f9204614c395904cf8b4299
I20260923 17:40:28.499208 12068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:40:28.507467 12068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:40:28 dist-test-slave-ppjl krb5kdc[11839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.124@KRBTEST.COM
Sep 23 17:40:28 dist-test-slave-ppjl krb5kdc[11839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185228, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.124@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
I20260923 17:40:28.520385 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.499525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45378 (local address 127.20.240.124:36689)
0923 17:40:28.500351 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:28.500363 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:28.500403 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:40:28.502036 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:28.502058 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:28.502459 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:28.502842 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:28.504306 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.505694 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.506962 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.507921 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.508139 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.512277 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:28.512286 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:28.512296 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:28.512340 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:28.515664 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.516526 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.516534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.516543 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.516876 (+ 333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.517274 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.517278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.517280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.517862 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:28.518206 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:28.518640 (+ 434us) server_negotiation.cc:299] Negotiation successful
0923 17:40:28.519099 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":350,"threads_started":1}
I20260923 17:40:28.527107 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.510340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.20.240.126:35711)
0923 17:40:28.510588 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:28.510596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:28.511440 (+ 844us) server_negotiation.cc:407] Connection header received
0923 17:40:28.511660 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:28.511666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:28.511759 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:28.511868 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:28.513239 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.514282 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.515894 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.516563 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.516657 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.519946 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:28.519953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:28.519958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:28.520004 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:28.524114 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.525021 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.525026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.525030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.525195 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.525533 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.525538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.525542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.525900 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:28.526066 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:28.526598 (+ 532us) server_negotiation.cc:299] Negotiation successful
0923 17:40:28.526767 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 17:40:28.527307 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.510345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.126:35711 (local address 127.0.0.1:56548)
0923 17:40:28.511281 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.511328 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:28.511525 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:28.511949 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:28.511955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:28.512713 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:28.513055 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.513069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.514478 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.514482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.515760 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.515778 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.516758 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.516764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.516938 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.517551 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:28.517569 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:28.519734 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:28.524266 (+ 4532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.524281 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.524288 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.524871 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.525294 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.525297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.525300 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.525426 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.526032 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:28.526046 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:28.526274 (+ 228us) client_negotiation.cc:769] Sending connection context
0923 17:40:28.526486 (+ 212us) client_negotiation.cc:240] Negotiation successful
0923 17:40:28.526825 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":462,"threads_started":1}
I20260923 17:40:28.528724 12068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:40:28 dist-test-slave-ppjl krb5kdc[11839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.126@KRBTEST.COM
Sep 23 17:40:28 dist-test-slave-ppjl krb5kdc[11839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185228, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.124@KRBTEST.COM for kudu/127.20.240.125@KRBTEST.COM
I20260923 17:40:28.540372 12073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.523772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.20.240.126:35711)
0923 17:40:28.524427 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:28.524434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:28.524457 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:40:28.524507 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:28.524512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:28.524597 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:28.524716 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:28.526154 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.527766 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.529118 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.529659 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.529767 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.534208 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:28.534213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:28.534216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:28.534249 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:28.537463 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.538339 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.538344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.538346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.538544 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.538960 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.538965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.538969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.539345 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:28.539621 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:28.539889 (+ 268us) server_negotiation.cc:299] Negotiation successful
0923 17:40:28.540070 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":311,"threads_started":1}
I20260923 17:40:28.543533 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.529352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.125:46573 (local address 127.0.0.1:53210)
0923 17:40:28.529524 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.529542 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:28.529643 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:28.530115 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:28.530118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:28.530439 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:28.530584 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.530592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.531926 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.531930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.532856 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.532868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.533715 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.533721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.533904 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.534496 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:28.534518 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:28.537078 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:28.540554 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.540560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.540564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.541090 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.541558 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.541563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.541567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.541727 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.542416 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:28.542421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:28.542705 (+ 284us) client_negotiation.cc:769] Sending connection context
0923 17:40:28.542827 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 17:40:28.543135 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 17:40:28.543582 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.529622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.20.240.125:46573)
0923 17:40:28.529788 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:28.529794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:28.529811 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:40:28.529868 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:28.529873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:28.529953 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:28.530090 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:28.530725 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.531763 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.533053 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.533568 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.533683 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.537316 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:28.537321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:28.537324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:28.537362 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:28.540378 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.541223 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.541227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.541230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.541364 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.541838 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.541843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.541846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.542183 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:28.542338 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:28.542963 (+ 625us) server_negotiation.cc:299] Negotiation successful
0923 17:40:28.543221 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:40:28.544451 12068 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:40:28 dist-test-slave-ppjl krb5kdc[11839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.240.125@KRBTEST.COM
Sep 23 17:40:28 dist-test-slave-ppjl krb5kdc[11839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185228, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.124@KRBTEST.COM for kudu/127.20.240.124@KRBTEST.COM
I20260923 17:40:28.559028 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.545242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45384)
0923 17:40:28.545491 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.545508 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:28.545614 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:28.545941 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:28.545950 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:28.546411 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:28.546593 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.546605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.547760 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.547764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.548660 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.548673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.549380 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.549386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.549600 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.550142 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:28.550160 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:28.552555 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:28.556255 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.556259 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.556263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.556742 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.557171 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.557176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.557180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.557336 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.557941 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:28.557946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:28.558138 (+ 192us) client_negotiation.cc:769] Sending connection context
0923 17:40:28.558295 (+ 157us) client_negotiation.cc:240] Negotiation successful
0923 17:40:28.558616 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260923 17:40:28.559252 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.545185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45384 (local address 127.20.240.124:36689)
0923 17:40:28.545339 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:28.545343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:28.545599 (+ 256us) server_negotiation.cc:407] Connection header received
0923 17:40:28.545739 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:28.545742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:28.545811 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:28.545891 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:28.546754 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.547618 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.548816 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.549217 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.549347 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.552725 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:28.552730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:28.552733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:28.552763 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:28.556119 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.556863 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.556868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.556872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.557052 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.557455 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.557458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.557460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.557817 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:28.557965 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:28.558399 (+ 434us) server_negotiation.cc:299] Negotiation successful
0923 17:40:28.558803 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":48}
I20260923 17:40:28.559264 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.542284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53212 (local address 127.20.240.125:46573)
0923 17:40:28.542790 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:28.542795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:28.542813 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:40:28.542849 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:28.542853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:28.542906 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:28.542998 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:28.544553 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.545695 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.547278 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.547702 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.547813 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.552513 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:28.552519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:28.552522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:28.552559 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:28.556047 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.556905 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.556911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.556914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.557056 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.557540 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.557544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.557548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.557847 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:28.558081 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:28.558399 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 17:40:28.558658 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":255,"threads_started":1}
I20260923 17:40:28.564201 21441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:40:28.564851 12068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d: Bootstrap starting.
I20260923 17:40:28.568156 12068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:28.569208 12068 log.cc:824] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:28.566643 11894 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "72aa1c828d004f10bad438fb32e0cb54"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56556
I20260923 17:40:28.572304 12068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d: No bootstrap required, opened a new log
I20260923 17:40:28.583186 12068 raft_consensus.cc:348] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } }
I20260923 17:40:28.583647 12068 raft_consensus.cc:729] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee89b2883f94d05b915688995be633d, State: Initialized, Role: FOLLOWER
I20260923 17:40:28.584116 12068 consensus_queue.cc:260] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } }
I20260923 17:40:28.585126 12076 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } } }
I20260923 17:40:28.585520 12076 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:40:28.586089 12068 sys_catalog.cc:564] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:40:28.595139 12087 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:40:28.595295 12087 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:40:28.593349 12001 master_runner.cc:186] Error getting master registration for 127.20.240.126:35711: OK, instance_id { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" instance_seqno: 1790185227473800 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:40:28.601312 12001 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:40:28.617400 11919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } attempt: 4
Sep 23 17:40:28 dist-test-slave-ppjl krb5kdc[11839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185227, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.126@KRBTEST.COM for kudu/127.20.240.124@KRBTEST.COM
I20260923 17:40:28.634420 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.618155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45396)
0923 17:40:28.618362 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.618383 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:28.618521 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:28.618957 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:28.618963 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:28.619521 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:28.619702 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.619713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.621139 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.621143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.622102 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.622115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.623204 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.623212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.623449 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.624155 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:28.624178 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:28.627086 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:28.631259 (+ 4173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.631265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.631270 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.631861 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.632297 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.632302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.632306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.632486 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.633268 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:28.633273 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:28.633494 (+ 221us) client_negotiation.cc:769] Sending connection context
0923 17:40:28.633661 (+ 167us) client_negotiation.cc:240] Negotiation successful
0923 17:40:28.633984 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260923 17:40:28.634423 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.618236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45396 (local address 127.20.240.124:36689)
0923 17:40:28.618481 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:28.618488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:28.618514 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:40:28.618669 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:28.618675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:28.618767 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:28.618921 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:28.619852 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.620982 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.622245 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.622980 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.623116 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.627329 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:28.627337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:28.627342 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:28.627388 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:28.631034 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.631998 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.632004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.632009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.632159 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.632621 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.632626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.632629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.633133 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:28.633312 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:28.633794 (+ 482us) server_negotiation.cc:299] Negotiation successful
0923 17:40:28.634034 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:40:28.641000 11919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54: Bootstrap starting.
I20260923 17:40:28.645393 11919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:28.646720 11919 log.cc:824] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:28.650210 11919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54: No bootstrap required, opened a new log
I20260923 17:40:28.657920 11919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } }
I20260923 17:40:28.658407 11919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72aa1c828d004f10bad438fb32e0cb54, State: Initialized, Role: FOLLOWER
I20260923 17:40:28.658951 11919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } }
I20260923 17:40:28.659970 12090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } } }
I20260923 17:40:28.660342 12090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:40:28.661353 11919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:28.661471 11894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 17:40:28.661665 11894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:28.664175 11894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } }
I20260923 17:40:28.665618 11894 leader_election.cc:288] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [CANDIDATE]: Term 1 election: Requested vote from peers b9256a28a7aa43679ddece9936d38503 (127.20.240.125:46573), aee89b2883f94d05b915688995be633d (127.20.240.124:36689)
I20260923 17:40:28.666992 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72aa1c828d004f10bad438fb32e0cb54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aee89b2883f94d05b915688995be633d"
I20260923 17:40:28.667675 12043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:28.671510 12043 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72aa1c828d004f10bad438fb32e0cb54 in term 1.
I20260923 17:40:28.672293 11868 leader_election.cc:302] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72aa1c828d004f10bad438fb32e0cb54, aee89b2883f94d05b915688995be633d; no voters:
I20260923 17:40:28.672829 12090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:40:28.674052 12090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [term 1 LEADER]: Becoming Leader. State: Replica: 72aa1c828d004f10bad438fb32e0cb54, State: Running, Role: LEADER
I20260923 17:40:28.674750 12090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } }
I20260923 17:40:28.685134 12094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72aa1c828d004f10bad438fb32e0cb54. Latest consensus state: current_term: 1 leader_uuid: "72aa1c828d004f10bad438fb32e0cb54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } } }
I20260923 17:40:28.685627 12094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:28.696586 12098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:40:28.705302 11994 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } attempt: 5
I20260923 17:40:28.709869 12098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260923 17:40:28.713795 11873 tablet.cc:2369] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:40:28.727519 12022 tablet.cc:2369] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:40:28.733994 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.693444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.20.240.124:36689)
0923 17:40:28.693628 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:28.693634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:28.713151 (+ 19517us) server_negotiation.cc:407] Connection header received
0923 17:40:28.715074 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:28.715081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:28.715191 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:28.715276 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:28.717718 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.718864 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.720914 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.721602 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.721714 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.725307 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:28.725314 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:28.725319 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:28.725360 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:28.730376 (+ 5016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.731279 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.731286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.731289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.731468 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.731952 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.731960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.731964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.732309 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:28.732562 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:28.733229 (+ 667us) server_negotiation.cc:299] Negotiation successful
0923 17:40:28.733464 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:40:28.743487 12073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.693587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.20.240.126:35711)
0923 17:40:28.693798 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:28.693804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:28.708368 (+ 14564us) server_negotiation.cc:407] Connection header received
0923 17:40:28.708862 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:28.708869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:28.708979 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:28.709061 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:28.712796 (+ 3735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.713922 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.722469 (+ 8547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.723151 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.723286 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.735525 (+ 12239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:28.735533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:28.735538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:28.735583 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:28.739214 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.740522 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.740528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.740532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.740689 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.741379 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.741384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.741387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.741748 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:28.741989 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:28.742855 (+ 866us) server_negotiation.cc:299] Negotiation successful
0923 17:40:28.743102 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 17:40:28 dist-test-slave-ppjl krb5kdc[11839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790185227, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.240.125@KRBTEST.COM for kudu/127.20.240.124@KRBTEST.COM
I20260923 17:40:28.748986 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.706948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.20.240.125:46573)
0923 17:40:28.707188 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:28.707195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:28.716012 (+ 8817us) server_negotiation.cc:407] Connection header received
0923 17:40:28.716654 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:28.716661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:28.716754 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:28.716830 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:28.724991 (+ 8161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.726019 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.732497 (+ 6478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.733169 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.733267 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.741438 (+ 8171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:28.741445 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:28.741450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:28.741492 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:28.745500 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.746580 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.746587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.746590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.746745 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.747209 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.747214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.747217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.747579 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:28.747821 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:28.748382 (+ 561us) server_negotiation.cc:299] Negotiation successful
0923 17:40:28.748626 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:40:28.754207 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.706020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.124:36689 (local address 127.0.0.1:45404)
0923 17:40:28.706447 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0923 17:40:28.706469 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:40:28.706597 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:40:28.736718 (+ 30121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:40:28.736724 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:40:28.737218 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:40:28.737421 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.737442 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.739111 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.739117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.740409 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:40:28.740433 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.741307 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.741313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:40:28.741522 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.742103 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:40:28.742126 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:40:28.744983 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:40:28.751632 (+ 6649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.751637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.751640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.752013 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.752490 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:40:28.752501 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:40:28.752505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:40:28.752634 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:40:28.753218 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:40:28.753223 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:40:28.753385 (+ 162us) client_negotiation.cc:769] Sending connection context
0923 17:40:28.753505 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 17:40:28.753788 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260923 17:40:28.754257 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:28.706831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.20.240.124:36689)
0923 17:40:28.725916 (+ 19085us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:28.725922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:28.725952 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:40:28.726007 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:28.726012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:28.726087 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:28.737581 (+ 11494us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:28.737817 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.738958 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.740606 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:28.741159 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:28.745677 (+ 4518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:28.745913 (+ 236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:40:28.745921 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:40:28.745926 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:40:28.745960 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:40:28.751413 (+ 5453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.752186 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.752191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.752194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.752368 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:40:28.752740 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:40:28.752745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:40:28.752749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:40:28.753091 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:40:28.753249 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:28.753611 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 17:40:28.753858 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18918,"thread_start_us":19348,"threads_started":1}
I20260923 17:40:28.759855 12114 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:40:28.760571 11994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503: Bootstrap starting.
I20260923 17:40:28.763489 12043 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d [term 1 FOLLOWER]: Refusing update from remote peer 72aa1c828d004f10bad438fb32e0cb54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:40:28.764904 11994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503: Neither blocks nor log segments found. Creating new log.
I20260923 17:40:28.764693 12094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:40:28.766248 11994 log.cc:824] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503: Log is configured to *not* fsync() on all Append() calls
I20260923 17:40:28.765676 11852 master_runner.cc:186] Error getting master registration for 127.20.240.125:46573: OK, instance_id { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" instance_seqno: 1790185227981099 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:40:28.777487 11852 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:40:28.781162 11994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503: No bootstrap required, opened a new log
I20260923 17:40:28.797499 11994 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } }
I20260923 17:40:28.798125 11994 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9256a28a7aa43679ddece9936d38503, State: Initialized, Role: FOLLOWER
I20260923 17:40:28.798774 11994 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } }
I20260923 17:40:28.798156 12076 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72aa1c828d004f10bad438fb32e0cb54. Latest consensus state: current_term: 1 leader_uuid: "72aa1c828d004f10bad438fb32e0cb54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } } }
I20260923 17:40:28.799093 12076 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:40:28.804478 12109 mvcc.cc:205] Tried to move back new op lower bound from 7332598696871079936 to 7332598696658698240. Current Snapshot: MvccSnapshot[applied={T|T < 7332598696871079936}]
I20260923 17:40:28.806195 12118 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } } }
I20260923 17:40:28.808756 12118 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:40:28.808619 11994 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:40:28.808992 11968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:40:28.809715 11969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72aa1c828d004f10bad438fb32e0cb54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b9256a28a7aa43679ddece9936d38503"
I20260923 17:40:28.814093 11968 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503 [term 1 FOLLOWER]: Refusing update from remote peer 72aa1c828d004f10bad438fb32e0cb54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:40:28.817902 11969 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72aa1c828d004f10bad438fb32e0cb54 in term 1.
I20260923 17:40:28.820689 12094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:40:28.821363 12094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72aa1c828d004f10bad438fb32e0cb54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } } }
I20260923 17:40:28.821617 12094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:28.827800 12076 sys_catalog.cc:455] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72aa1c828d004f10bad438fb32e0cb54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } } }
I20260923 17:40:28.828104 12076 sys_catalog.cc:458] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:40:28.829963 12098 catalog_manager.cc:1347] Generated new cluster ID: c6436d0c949d4a8eb1a1d6e663cd74fa
I20260923 17:40:28.830084 12098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:40:28.859318 12098 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:40:28.861361 12098 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:40:28.861737 12118 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72aa1c828d004f10bad438fb32e0cb54. Latest consensus state: current_term: 1 leader_uuid: "72aa1c828d004f10bad438fb32e0cb54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } } }
I20260923 17:40:28.862071 12118 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:40:28.864339 12090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72aa1c828d004f10bad438fb32e0cb54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } } }
I20260923 17:40:28.864243 12129 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72aa1c828d004f10bad438fb32e0cb54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72aa1c828d004f10bad438fb32e0cb54" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 35711 } } peers { permanent_uuid: "b9256a28a7aa43679ddece9936d38503" member_type: VOTER last_known_addr { host: "127.20.240.125" port: 46573 } } peers { permanent_uuid: "aee89b2883f94d05b915688995be633d" member_type: VOTER last_known_addr { host: "127.20.240.124" port: 36689 } } }
I20260923 17:40:28.864626 12090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54 [sys.catalog]: This master's current role is: LEADER
I20260923 17:40:28.864634 12129 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:40:28.896277 12136 catalog_manager.cc:1259] Loaded cluster ID: c6436d0c949d4a8eb1a1d6e663cd74fa
I20260923 17:40:28.896503 12136 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503: loading cluster ID for follower catalog manager: success
I20260923 17:40:28.901104 12136 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503: acquiring CA information for follower catalog manager: success
W20260923 17:40:28.908957 12136 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260923 17:40:28.919324 12098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72aa1c828d004f10bad438fb32e0cb54: Generated new TSK 0
I20260923 17:40:28.920614 12098 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:40:29.054260 12073 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:29.041499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.20.240.126:35711)
0923 17:40:29.041701 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:29.041709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:29.051912 (+ 10203us) server_negotiation.cc:407] Connection header received
0923 17:40:29.052277 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:29.052285 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:29.052394 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:29.052474 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:29.053942 (+ 1468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56566: BlockingRecv error: recv got EOF from 127.0.0.1:56566 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260923 17:40:29.060750 12105 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:29.047237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.20.240.124:36689)
0923 17:40:29.047420 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:29.047427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:29.056582 (+ 9155us) server_negotiation.cc:407] Connection header received
0923 17:40:29.058120 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:29.058127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:29.058243 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:29.058340 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:29.060487 (+ 2147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45408: BlockingRecv error: recv got EOF from 127.0.0.1:45408 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260923 17:40:29.061959 12074 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:29.057192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53236 (local address 127.20.240.125:46573)
0923 17:40:29.057396 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:29.057402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:29.060134 (+ 2732us) server_negotiation.cc:407] Connection header received
0923 17:40:29.060414 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:29.060420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:29.060515 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:29.060587 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:29.061646 (+ 1059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53236: BlockingRecv error: recv got EOF from 127.0.0.1:53236 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 17:40:29.103765 12074 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:29.093328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.20.240.125:46573)
0923 17:40:29.093550 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:29.093557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:29.101173 (+ 7616us) server_negotiation.cc:407] Connection header received
0923 17:40:29.101865 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:29.101873 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:29.102066 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:29.102164 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:29.103484 (+ 1320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53242: BlockingRecv error: recv got EOF from 127.0.0.1:53242 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260923 17:40:29.153666 12074 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:29.136449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.20.240.125:46573)
0923 17:40:29.136671 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:29.136679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:29.151849 (+ 15170us) server_negotiation.cc:407] Connection header received
0923 17:40:29.152105 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:29.152112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:29.152222 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:29.152305 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:29.153375 (+ 1070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53258: BlockingRecv error: recv got EOF from 127.0.0.1:53258 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 17:40:29.203589 12074 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:29.193356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53266 (local address 127.20.240.125:46573)
0923 17:40:29.193572 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:29.193579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:29.201149 (+ 7570us) server_negotiation.cc:407] Connection header received
0923 17:40:29.201795 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:29.201803 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:29.201922 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:29.202032 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:29.203289 (+ 1257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53266: BlockingRecv error: recv got EOF from 127.0.0.1:53266 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260923 17:40:29.249209 12074 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:29.239205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.20.240.125:46573)
0923 17:40:29.239470 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:29.239477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:29.247105 (+ 7628us) server_negotiation.cc:407] Connection header received
0923 17:40:29.247667 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:29.247674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:29.247781 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:29.247862 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:29.248943 (+ 1081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53278: BlockingRecv error: recv got EOF from 127.0.0.1:53278 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
W20260923 17:40:29.293431 12074 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:29.290852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.20.240.125:46573)
0923 17:40:29.291085 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:29.291094 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:29.291873 (+ 779us) server_negotiation.cc:407] Connection header received
0923 17:40:29.292103 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:29.292109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:29.292218 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:29.292315 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:29.293200 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53284: BlockingRecv error: recv got EOF from 127.0.0.1:53284 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260923 17:40:29.314904 12074 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:29.312714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.20.240.125:46573)
0923 17:40:29.312957 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:29.312964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:29.313477 (+ 513us) server_negotiation.cc:407] Connection header received
0923 17:40:29.313689 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:29.313694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:29.313768 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:29.313887 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:29.314624 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53294: BlockingRecv error: recv got EOF from 127.0.0.1:53294 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260923 17:40:29.352655 12074 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:29.350300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.20.240.125:46573)
0923 17:40:29.350535 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:29.350542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:29.351097 (+ 555us) server_negotiation.cc:407] Connection header received
0923 17:40:29.351346 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:29.351351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:29.351432 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:29.351518 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:29.352412 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53310: BlockingRecv error: recv got EOF from 127.0.0.1:53310 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260923 17:40:29.422329 12074 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:29.420358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.20.240.125:46573)
0923 17:40:29.420577 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:29.420583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:29.420863 (+ 280us) server_negotiation.cc:407] Connection header received
0923 17:40:29.421093 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:29.421099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:29.421212 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:29.421337 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:29.422081 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53314: BlockingRecv error: recv got EOF from 127.0.0.1:53314 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260923 17:40:29.556006 12074 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:29.553976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53328 (local address 127.20.240.125:46573)
0923 17:40:29.554188 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:29.554195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:29.554534 (+ 339us) server_negotiation.cc:407] Connection header received
0923 17:40:29.554779 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:29.554785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:29.554886 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:29.554974 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:29.555776 (+ 802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53328: BlockingRecv error: recv got EOF from 127.0.0.1:53328 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:40:29.597493 12087 catalog_manager.cc:1259] Loaded cluster ID: c6436d0c949d4a8eb1a1d6e663cd74fa
I20260923 17:40:29.597615 12087 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d: loading cluster ID for follower catalog manager: success
I20260923 17:40:29.600457 12087 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d: acquiring CA information for follower catalog manager: success
I20260923 17:40:29.602497 12087 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aee89b2883f94d05b915688995be633d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 17:40:29.817570 12074 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:40:29.815695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.20.240.125:46573)
0923 17:40:29.815908 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:29.815915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:29.816018 (+ 103us) server_negotiation.cc:407] Connection header received
0923 17:40:29.816263 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:29.816267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:29.816368 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:29.816471 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:40:29.817297 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53344: BlockingRecv error: recv got EOF from 127.0.0.1:53344 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:40:29.911907 12136 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b9256a28a7aa43679ddece9936d38503: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:40:30.339571 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:30.333053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.20.240.125:46573)
0923 17:40:30.333534 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:30.333538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:30.333665 (+ 127us) server_negotiation.cc:407] Connection header received
0923 17:40:30.333903 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:30.333907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:30.334061 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:30.334171 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:30.334619 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:30.335460 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:30.337048 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:30.337498 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:30.337610 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:30.338219 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:30.338835 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:30.338962 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:30.339118 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 17:40:30.339239 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":242,"threads_started":1}
I20260923 17:40:30.351222 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:30.345371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45420 (local address 127.20.240.124:36689)
0923 17:40:30.345872 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:30.345879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:30.345920 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:40:30.346161 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:30.346165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:30.346317 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:30.346483 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:30.346885 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:30.347710 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:30.349016 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:30.349420 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:30.349534 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:30.350048 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:30.350596 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:30.350740 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:30.350816 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 17:40:30.350903 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":236,"threads_started":1}
I20260923 17:40:30.364905 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:30.358240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53368 (local address 127.20.240.125:46573)
0923 17:40:30.358454 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:30.358460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:30.359122 (+ 662us) server_negotiation.cc:407] Connection header received
0923 17:40:30.359388 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:30.359394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:30.359473 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:30.359544 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:30.360126 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:30.361155 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:30.362786 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:30.363424 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:30.363556 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:30.364137 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:30.364273 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:30.364364 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:30.364511 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 17:40:30.364603 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:40:30.365043 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:30.357216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.20.240.124:36689)
0923 17:40:30.357431 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:30.357438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:30.358563 (+ 1125us) server_negotiation.cc:407] Connection header received
0923 17:40:30.358874 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:30.358880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:30.358955 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:30.359008 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:30.359842 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:30.360918 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:30.363010 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:30.363666 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:30.363766 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:30.364321 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:30.364478 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:30.364556 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:30.364744 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 17:40:30.364821 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:40:30.365378 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:40:30.357096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.20.240.126:35711)
0923 17:40:30.357565 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0923 17:40:30.357571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:40:30.358198 (+ 627us) server_negotiation.cc:407] Connection header received
0923 17:40:30.358658 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:40:30.358665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:40:30.358906 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:40:30.359118 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:40:30.359730 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:30.360608 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:30.362703 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:40:30.363326 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:40:30.363449 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:40:30.363967 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:40:30.364437 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:40:30.364930 (+ 493us) server_negotiation.cc:1035] Waiting for connection context
0923 17:40:30.364984 (+ 54us) server_negotiation.cc:299] Negotiation successful
0923 17:40:30.365075 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":212,"threads_started":1}
I20260923 17:40:30.371466 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 11852
I20260923 17:40:30.372583 11914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:40:30.484858 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 11852
W20260923 17:40:30.502429 12017 connection.cc:535] server connection from 127.0.0.1:45396 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260923 17:40:30.502422 11942 connection.cc:535] server connection from 127.0.0.1:53208 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 17:40:30.503197 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 11926
I20260923 17:40:30.504312 11989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:40:30.610663 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 11926
I20260923 17:40:30.627633 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu pid 12001
I20260923 17:40:30.628778 12063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:40:30.735714 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSmAN3S/build/asan/bin/kudu with pid 12001
2026-09-23T17:40:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4996 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9276 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (364719 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST